Commit Graph
195 Commits
Author SHA1 Message Date
asepharyana 2653e23283 test(e2e): auth headers on web API uploads + 1GB large-file suite
- Web API upload/delete tests now send ADMIN_API_TOKEN (POST/DELETE require auth)
- New 1GB suite: S3 single PUT, S3 multipart (40MB parts), ListObjects verify,
  POST /api/upload; skips without S3_SECRET_KEY + ADMIN_API_TOKEN
- Fix: Bun.write accepts the Response directly (dl.stream() doesn't exist)
2026-08-01 20:33:10 +07:00
asepharyana 91ec588a88 feat(web): public read-only file browser — GET API public, writes require admin
- /api/v1/* GET (list buckets/objects, download) no longer requires auth
- POST/DELETE/PUT stay behind requireAuth (upload, create/delete bucket, copy, delete object)
- FE drops blocking login screen: visitors browse + download freely
- Admin-only UI (create bucket, upload dropzone, delete, S3 creds) hidden in read-only mode
- Login button in topbar to unlock admin actions
2026-08-01 20:31:40 +07:00
asepharyana 864d41d8fc fix(nix): restrict flake to x86_64-linux (nixpkgs 26.11 dropped darwin) 2026-08-01 18:03:38 +07:00
asepharyana c14be68ff8 ci: publish flake to FlakeHub (rolling) 2026-08-01 17:58:16 +07:00
asepharyana 811a68821d ci: migrate CI to GitHub Actions (deploy nix + mirror ke Gitea backup)
Mirror to Gitea / mirror (push) Successful in 22s
Build & Deploy (Nix) / build-and-deploy (push) Failing after 35m18s
2026-08-01 16:38:58 +07:00
asepharyana bf616f6790 fix: root page 500 — resolve home.html for dev & prod layouts
Build & Deploy (Nix) / build-and-deploy (push) Successful in 54s
handleHome looked up `${import.meta.dir}/home.html` which exists in neither
layout: dev (src/interfaces/http/controllers/) nor prod bundle
($out/share/teleuploader/dist/ — flake copies home.html beside dist/).

Add resolveHomeHtml(): walk up from import.meta.dir (bounded) to find
home.html. Works for dev (src/home.html, 4 levels up) and prod
(../home.html, 1 level up). Fails fast with a clear error instead of a
bare ENOENT 500. Tests cover both layouts + not-found fallback.
2026-08-01 13:15:10 +07:00
asepharyana 24dfb1c6b1 fix: fail-fast guard TELEGRAM_CHUNK_SIZE_BYTES <= 19MB (Telegram getFile limit 20MB)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 56s
Chunk parts > 19MB are stored to Telegram but getFile cannot resolve files over 20MB ('Bad Request: file is too big'), making every part undownloadable (prod bug 2026-08-01: 48MB chunk -> download 500).

- src/env.ts: reject TELEGRAM_CHUNK_SIZE_BYTES > 19922944 at startup (log error + throw), default changed 20MB -> 19MB
- src/shared/utils/validation.ts: TELEGRAM_CHUNK_SIZE_MAX_BYTES constant; asSafeChunkSize now enforces the max at runtime (covers S3 multipart parts too)
- test/env.test.ts: unit tests + subprocess fail-fast tests (48MB rejected, 19MB accepted)
- test/helpers/setup-env.ts: pin safe chunk size so a stale .env can't break the suite
- .env.example + CLAUDE.md: document the 20MB getFile limit
2026-08-01 13:06:42 +07:00
Claude b9a3fcd828 ci: nix build in CI + profile-based deploy (auto-update systemd)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 49s
2026-07-30 16:27:35 +07:00
Claude f00b944e56 fix: nix build di CI — init none + flakes enabled
Build & Deploy (Nix) / build-and-deploy (push) Successful in 49s
2026-07-30 16:23:15 +07:00
Claude 73b0630cbd v1.1.0 + ci: build di CI, nix copy ke VPS (ringan, gak build di target)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 1m0s
2026-07-30 16:13:59 +07:00
Claude d1771720bd ci: build di VPS target via SSH (seperti yang sudah terverifikasi)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 12s
2026-07-30 16:08:13 +07:00
Claude 7dd2a4f0c7 fix: build di CI, nix copy ke VPS target (ringan)
Build & Deploy (Nix) / build-and-deploy (push) Failing after 56s
2026-07-30 16:05:20 +07:00
Claude fbc1f1822a fix: deploy via SSH, build + deploy on VPS directly
Build & Deploy (Nix) / build-and-deploy (push) Successful in 15s
The CI runner container can't access the VPS host's systemd and Nix
directly. Instead of installing Nix in the container and trying to
access the host, SSH directly to the VPS to build and deploy.

This approach:
1. SSHs to the VPS using the VPS_SSH_KEY_VALUE secret
2. Pulls the latest code on the VPS
3. Builds with Nix directly on the VPS
4. Updates nix-env profile and restarts systemd service

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:43:14 +07:00
Claude c768b29e9d fix: deploy via Docker socket with host filesystem access
Build & Deploy (Nix) / build-and-deploy (push) Failing after 53s
Docker socket /var/run/docker.sock is available in the runner
container. Use docker run --pid=host --privileged to access
the VPS host filesystem via chroot to execute nix-env and systemctl.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:39:38 +07:00
Claude 59535d66b8 fix: add environment inspect step before deploy
Build & Deploy (Nix) / build-and-deploy (push) Failing after 46s
Diagnosing deploy failure: systemctl unavailable inside runner container.
Adding inspect step to understand available mounts and access mechanisms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:35:47 +07:00
Claude 319d2980c5 fix: use full path for nix-env in deploy step
Build & Deploy (Nix) / build-and-deploy (push) Failing after 47s
The deploy step runs with 'sudo' which resets PATH, so nix-env is
not found. Use explicit path to the nix-env binary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:33:29 +07:00
Claude 4acfa3d2e9 fix: use Determinate Systems nix installer without --no-daemon
Build & Deploy (Nix) / build-and-deploy (push) Failing after 52s
Re-instate Determinate Systems installer with correct flags:
- install --no-confirm (not --no-daemon which it doesn't support)
- Nix installs to /nix/var/nix/profiles/default/bin
- Source daemon profile in build step

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:32:09 +07:00
Claude 5bd3d286ce fix: use Nix official installer with --no-daemon
Build & Deploy (Nix) / build-and-deploy (push) Failing after 11s
The Determinate Systems installer doesn't support --no-daemon.
Switch to the official Nix installer which has a well-documented
--no-daemon flag suitable for container/CI use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:30:59 +07:00
Claude bfafcc0a20 fix: install Nix in CI before building
Build & Deploy (Nix) / build-and-deploy (push) Failing after 7s
The Gitea Actions runner image (docker.gitea.com/runner-images:ubuntu-latest)
does not have Nix pre-installed, causing 'nix: command not found' on the
build step.

Add an 'Install Nix' step using the Determinate Systems installer
(--no-daemon mode since the runner runs in a container).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:29:46 +07:00
Claude c26ca806d0 fix: update CI to Nix build + systemd deploy (no Docker)
Build & Deploy (Nix) / build-and-deploy (push) Failing after 5s
2026-07-30 15:08:23 +07:00
Claude bc3d1f2e3f chore: trigger CI
Deploy FileDrop / deploy (push) Successful in 1m35s
2026-07-30 14:37:10 +07:00
Claude f95022142d feat: add Nix flake for TeleUploader build
Deploy FileDrop / deploy (push) Waiting to run
- flake.nix with Bun build from nixpkgs (1.3.x)
- flake.lock pinned inputs
- .gitignore result symlink from nix build

Build: nix build .#teleuploader --impure --option sandbox false
Until CI/CD with binary cache is set up
2026-07-30 13:00:53 +07:00
Claude bd766792b1 a
Deploy FileDrop / deploy (push) Failing after 14m44s
2026-07-29 19:25:31 +07:00
Claude d22f788e3e fix: wrap Bun.file().stream() in Readable.from() for Telegraf compat
Deploy FileDrop / deploy (push) Failing after 20s
- Replaces createReadStream / bare ReadableStream with
  Readable.from(Bun.file(path).stream()) — works in both
  test (Bun.write + Bun.file) and production (Node Readable)
- Reverts writeBufferToTemp back to Bun.write
2026-07-29 17:26:51 +07:00
Claude 1ecab987a6 fix: send non-JPEG images as document instead of photo; fix Dockerfile home.html path
Deploy FileDrop / deploy (push) Failing after 23s
- getFileType: only classify image/jpeg as 'photo'; png/gif/webp → 'document'
  (Telegram Bot API rejects non-JPEG for sendPhoto)
- Dockerfile: copy home.html to dist/ instead of root (import.meta.dir = dist/)
- Update test assertion for getFileType(image/png) → 'document'
2026-07-29 17:15:07 +07:00
Claude 9a4853a484 fix: remove UploadBatcher crash window, make bot concurrency configurable, fix all test import paths
Deploy FileDrop / deploy (push) Successful in 48s
- Removed UploadBatcher (src/infrastructure/telegram/upload-batcher.ts + DI):
  pending uploads no longer lost on crash, files sent directly to Telegram
- Changed upload-controller to use Bun.file().stream() instead of createReadStream
- Made PER_BOT_CONCURRENCY configurable via TELEGRAM_BOT_CONCURRENCY env
- Fixed 18 test files with updated import paths and mock shapes
- Updated package.json test script: telegramQueue.test.ts → bot-pool.test.ts
- Build, lint, and test suite all pass
2026-07-29 17:06:35 +07:00
Claude ad917f6675 refactor: dedup dead code, shared utils, conditional headers helper
- Extract shared utils: asSafeChunkSize (validation.ts), S3 detection (s3-detection.ts)
- Remove dead _handleMaybeS3Root from index.ts and routes/index.ts
- Remove dead _asArray from file-controller.ts
- Consolidate maybeCompressChunk into shared compress.ts
- Extract checkConditionalHeaders helper, remove ~120 lines dupe in s3-controller
- Remove 500+ lines dead code from s3-object.ts (unused use cases + helpers)
- Delegate upload-file.ts chunked path to ChunkedStorage, remove dupe
- Fix broken dynamic import in file-controller.ts → proper DI
- Fix test/files.test.ts import path and mocks

[skip ci]
2026-07-29 16:39:52 +07:00
Claude fab91ad69c feat: dedup wave 3 — constant + streamToTemp utility
Deploy FileDrop / deploy (push) Successful in 44s
- ♻️ randomUUID() → nanoid() di s3-object.ts (eliminasi node:crypto)
-  DEFAULT_FILE_TYPE constant, ganti 8× hardcoded 'document'
-  streamToTemp() shared utility (src/shared/utils/temp-stream.ts)
- ♻️ 3× streaming-to-temp pattern di s3-controller, upload-controller,
  web-api-controller → pake streamToTemp()
- Lint  Build 
2026-07-29 16:10:49 +07:00
Claude 3501d547c0 feat: dedup wave 2 — file factory + V1/V2 mapping + bot handler
Deploy FileDrop / deploy (push) Successful in 42s
-  buildNewFile() factory (src/domain/entities/file-factory.ts)
  eliminates ~380 lines of archive null / isDeleted / field defaults
- ♻️ 27 creation blocks → buildNewFile() across 8 files
- ♻️ S3 V1/V2 object listing mapping → mapFileToListEntry()
- ♻️ Bot handler size validation → checkFileSize() from shared utils
- 🔎 computeHash audit: all 10 instances are streaming (no dedup)
- Lint  Build  Push 
2026-07-29 16:02:01 +07:00
Claude 332853f398 chore: dedup dead code + compress utility
Deploy FileDrop / deploy (push) Successful in 43s
- Hapus src/shared/utils/retry.ts (0 imports, dead)
- Hapus src/shared/errors/index.ts (7 classes, 0 imports)
- Hapus drizzle type exports dari schema.ts (tidak dipake)
- Hapus src/config/index.ts, redirect 8 imports langsung ke env.ts
- Ekstrak maybeCompressChunk ke shared/utils/compress.ts
- Hapus duplikasi gzipSync di upload-file.ts + chunked-storage.ts
- Lint clean, build clean
2026-07-29 15:51:39 +07:00
Claude 4c216b1d9f refactor: full DDD + Clean Architecture refactor
Deploy FileDrop / deploy (push) Successful in 48s
- Hapus src/utils/ (17 files) + src/db/ (8 files) dead code
- Absorb 7 re-export stubs → real impl di lokasi DDD
- Buat src/infrastructure/di.ts (DI container)
- Rewrite 5 controllers pakai repository/DI
- Fix shared/utils imports, env.ts, routes, index.ts
- Update package.json build path migrate
- Lint clean, build clean
2026-07-29 15:41:50 +07:00
Claude 3580001b8a fix: add backward compat env vars to docker-compose for Gitea CI
Deploy FileDrop / deploy (push) Successful in 27s
2026-07-29 15:08:49 +07:00
Claude 6d3696d261 fix: handler.ts uses config.botTokens[0] instead of removed config.botToken
Deploy FileDrop / deploy (push) Successful in 27s
2026-07-29 15:08:23 +07:00
Claude 1484d5265d refactor: merge BOT_TOKEN + ADDITIONAL_BOT_TOKENS into single BOT_TOKENS env + speed audit
Deploy FileDrop / deploy (push) Failing after 19s
BOT_TOKENS env:
- Single BOT_TOKENS env var (comma-separated) replaces BOT_TOKEN + ADDITIONAL_BOT_TOKENS
- Backward compat: falls back to BOT_TOKEN + ADDITIONAL_BOT_TOKENS if BOT_TOKENS unset
- Config exposes botTokens: string[] instead of botToken + additionalBotTokens
- Updated env.ts, bot-pool.ts, docker-compose.yml, .env.example, CLAUDE.md, all tests

Speed audit (S3 -> Telegram upload flow):
- Hoisted 5 dynamic await import('../../../db/index') to top-level static imports
  in s3-controller.ts (3x) and web-api-controller.ts (2x)
  -> saves module resolution + async overhead on every upload
- Removed stale UPLOAD_CONCURRENCY env from docker-compose.yml
  (already removed from env.ts in prior refactor)

Upload flow is already concurrent:
- streamBodyToTemp() uses Bun.file(path).writer() — O(1) memory, safe for multi-GB blobs
- utils/chunked-storage.ts reads chunks serially but uploads concurrently with
  inFlight backpressure at effectiveConcurrency * 2 (= 16 with 8 bots)
- bot-pool.ts: per-bot PQueue(concurrency=1), 8 bots = 8 concurrent uploads per file,
  selectBot() picks least-loaded, 429 detection + inner+outer retry loops
- TELEGRAM_API_TIMEOUT_MS=120s — ample for 48MB chunks
- Infrastructure chunked-storage.ts (DI-based, dead code) has serial upload trap —
  noted for future cleanup
2026-07-29 15:03:02 +07:00
Claude 245ea169ad chore: add 2 bot tokens, update PRODUCTION_ENV for 6 bots
Deploy FileDrop / deploy (push) Successful in 52s
- Now 6 total bots: 1 main + 5 additional
- Removed UPLOAD_CONCURRENCY from PRODUCTION_ENV

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 14:48:50 +07:00
Claude 2cf6aa3275 refactor: implement per-bot queue architecture
Deploy FileDrop / deploy (push) Successful in 45s
- Each bot has its own PQueue with concurrency=1
- selectBot() assigns uploads to least-loaded available bot
- 429 rate limits are tracked per-bot with cooldown timers
- Failed uploads retry on next available bot
- Removed global upload-queue.ts and uploadConcurrency config
- Updated ITelegramService interface
2026-07-29 13:36:16 +07:00
Claude d0de4de2d5 refactor: remove uploadConcurrency from config
Effective concurrency derived from bot pool size. Chunked-storage
backpressure now uses botPool.getEffectiveConcurrency().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:30:49 +07:00
Claude adbf9b5efa refactor: remove global upload queue
Per-bot queues now handle concurrency internally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:28:38 +07:00
Claude a986ce1e08 fix: per-bot queue fixes — outer loop transient retry, getFileInfo logging, test file, safety net comment, empty-bot guard
- Add MAX_OUTER_RETRIES constant and transientAttempts counter for outer-loop retry
- Restore getFileInfo transient retry logging with bot identity and fileId
- Create test/bot-pool.test.ts with 4 tests for core BotPool behavior
- Add empty-bots guard in selectBot() returning null
- Add safety net comment and improved logging for outer 429 catch

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:25:03 +07:00
Claude 5617d0ff35 refactor: per-bot queue with selectBot() and rate-limit tracking
Each bot has its own PQueue (concurrency=1). Uploads are assigned to
the least-loaded available bot. On 429, the bot is marked rate-limited
and the upload retries on the next available bot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:17:49 +07:00
Claude d7d6ae0f0d refactor: remove enqueueUpload from ITelegramService and BotPool
Per-bot queue replaces global upload queue — BotPool handles
queueing internally.

- Remove enqueueUpload method signature from ITelegramService interface
- Remove enqueueUpload method from BotPool class
- Remove import of enqueueUpload from upload-queue module
- Refactor forwardToStorage to call executeWithBotRetry directly
  instead of wrapping via enqueueUpload
- Fix trailing blank lines flagged by Biome formatter

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:14:26 +07:00
Claude ea31c4c591 docs: add per-bot queue design spec
Per-bot queue architecture for Telegram upload rate-limit safety.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:09:34 +07:00
Claude 996b7d06ef chore: add more bot tokens for better rate limit distribution
Deploy FileDrop / deploy (push) Successful in 41s
Previous: 4 bots → now: 6 bots
Each bot handles fewer concurrent uploads, reducing 429 rate limits
and avoiding thundering-herd sleep patterns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:02:00 +07:00
Claude 3f541121f5 fix: yield microtask after Promise.race in backpressure check
Deploy FileDrop / deploy (push) Successful in 48s
Adds a setTimeout(0) microtask yield after Promise.race to ensure
the .finally() handler that removes promises from the inFlight
set has executed before the next backpressure check.

Also ensure parts array is sorted by partNumber after concurrent
uploads complete, since promises resolve in arbitrary order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 12:44:14 +07:00
Claude 1a38b32fbc feat: concurrent chunk uploads within single file
Deploy FileDrop / deploy (push) Successful in 41s
Previously uploadFileInTelegramChunks awaited each chunk's upload
before reading the next, making all chunks sequential within a file.
Now chunks are uploaded concurrently using a managed Set of in-flight
promises with backpressure limiting (2x uploadConcurrency).

This means a single 1GB Docker layer split into 48MB chunks will
have up to 32 chunks uploading simultaneously, not one at a time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 12:28:05 +07:00
Claude b034c3131d chore: trigger deploy for TELEGRAM_CHUNK_SIZE_BYTES=48MB
Deploy FileDrop / deploy (push) Successful in 52s
2026-07-29 12:23:44 +07:00
Claude d8b8a1381d chore: increase Telegram chunk size to 48MB (max ~49MB)
Deploy FileDrop / deploy (push) Successful in 44s
Tested actual Telegram Bot API limit:
- 49MB 
- 50MB  (413 Request Entity Too Large)
Set TELEGRAM_CHUNK_SIZE_BYTES=50331648 (48MB) for safety

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:57:46 +07:00
Claude b0c5327bb3 fix: add content_type column to multipart_uploads migration
Deploy FileDrop / deploy (push) Successful in 50s
The createMultipartUpload function inserts content_type but the
database column was missing, causing 500 errors on every Gitea
Docker registry push (which uses multipart uploads for blob storage).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:09:24 +07:00
Claude e3e7430ce0 fix: zero-downtime deploy with --wait flag + fallback
Deploy FileDrop / deploy (push) Canceled after 0s
2026-07-29 09:57:47 +07:00
Claude d8b1878a75 fix: add ETag and headers to 304 Not Modified responses
Deploy FileDrop / deploy (push) Failing after 14m18s
AWS SDK requires ETag header in 304 responses. Without it, the SDK
throws UnknownError despite receiving a valid 304 status code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:23:16 +07:00