TeleUploader/docs/superpowers/specs/2026-07-06-s3-compatible-teleuploader-design.md

S3-Compatible TeleUploader + Web File Manager

Date: 2026-07-06 Status: Approved

Objective: Transform TeleUploader into an S3-compatible storage server (Telegram-backed) with a web file manager UI. Existing upload/download API paths remain unchanged.

---

1. Architecture Overview

S3 Client (aws-cli, rclone, s3cmd, MinIO Client)
         │
         ▼  AWS SigV4 + XML
┌─────────────────────────────────────┐
│          S3 Protocol Dispatcher     │  src/routes/s3.ts
│  - Parses path, query, auth headers │
│  - Routes to S3 operation handlers  │
├─────────────────────────────────────┤
│   JSON API v1 (for Web UI)          │  src/routes/web-api.ts
│  - Bucket CRUD                      │
│  - Object list/upload/copy/delete   │
├─────────────────────────────────────┤
│         Web File Manager            │  src/routes/home.ts + home.html
│  - Bucket browser, upload/download  │
├─────────────────────────────────────┤
│       Database Layer                │  src/db/buckets.ts
│  - buckets, files (extended),       │  src/db/multipart.ts
│  - multipart_uploads, parts         │
├─────────────────────────────────────┤
│       Telegram Storage Layer        │  src/utils/telegram.ts (existing)
│  - Single Telegram channel           │
│  - Files stored as Telegram docs    │
└─────────────────────────────────────┘

Component Responsibilities

Component	Responsibility
S3 Protocol Dispatcher	S3 fallback route ($). Parse Auth header, detect SigV4, route by method+path+query
S3 Operations	~20 operations (ListBuckets, GetObject, PutObject, ListObjectsV2, MultipartUpload, DeleteObjects, etc)
Auth (SigV4)	Verify AWS SigV4 signatures, reject invalid/missing with 403 XML error
XML Builder	Template functions to construct S3 XML responses. Parse incoming XML (DeleteObjects body)
JSON API v1	JSON wrapper for web UI to avoid XML parsing in browser
Web File Manager	Single-page app served at / with bucket browser, upload, delete, search
DB Layer	New tables: buckets, multipart_uploads, multipart_parts. Extended files table
Telegram Storage	Existing forwardToStorage/getFileInfo; unchanged

Routing Priority

Route matching order (existing unchanged, S3 added as catch-all):

Priority	Route	Handler
1	/api/upload	Existing upload handler
2	/f/:public_id	Existing file redirect
3	/file/:public_id/info	Existing file info
4	/health	Existing health
5	/docs / /swagger.json	Existing swagger
6	/	Web file manager UI (NEW)
7	/api/v1/*	JSON API for Web UI (NEW)
8	$ (catch-all)	S3 dispatcher (NEW)

The catch-all route ($ in Bun.serve()) inspects the Authorization header:

• Contains AWS4-HMAC-SHA256 → handle as S3 request
• Otherwise → 404

---

2. Database Schema

2.1 buckets Table (NEW)

CREATE TABLE buckets (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT UNIQUE NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

-- Bucket name constraints: lowercase, no underscore, 3-63 chars (S3 spec)

2.2 files Table — Extended

New columns added alongside existing ones:

ALTER TABLE files ADD COLUMN bucket_id UUID REFERENCES buckets(id);
ALTER TABLE files ADD COLUMN s3_key TEXT;           -- e.g. "images/logo.png"
ALTER TABLE files ADD COLUMN storage_backend TEXT DEFAULT 'telegram';
ALTER TABLE files ADD COLUMN is_deleted BOOLEAN DEFAULT false;
ALTER TABLE files ADD COLUMN multipart_upload_id TEXT;

CREATE UNIQUE INDEX idx_files_bucket_key ON files(bucket_id, s3_key) WHERE is_deleted = false;
CREATE INDEX idx_files_bucket_prefix ON files(bucket_id, s3_key text_pattern_ops);

• s3_key is the full object key path
• is_deleted enables soft-delete for S3 DeleteObject
• multipart_upload_id links parts to their parent completed multipart object
• public_id (existing) remains primary identifier for Telegram redirect
• Existing non-S3 uploads have bucket_id = NULL, s3_key = NULL

2.3 multipart_uploads Table (NEW)

CREATE TABLE multipart_uploads (
  upload_id TEXT PRIMARY KEY,
  bucket_id UUID NOT NULL REFERENCES buckets(id),
  s3_key TEXT NOT NULL,
  initiated_at TIMESTAMPTZ DEFAULT now(),
  status TEXT DEFAULT 'in_progress' CHECK (status IN ('in_progress', 'completed', 'aborted')),
  initiated_by TEXT
);

2.4 multipart_parts Table (NEW)

CREATE TABLE multipart_parts (
  id SERIAL PRIMARY KEY,
  upload_id TEXT NOT NULL REFERENCES multipart_uploads(upload_id) ON DELETE CASCADE,
  part_number INT NOT NULL CHECK (part_number BETWEEN 1 AND 10000),
  telegram_file_id TEXT NOT NULL,
  telegram_file_unique_id TEXT NOT NULL,
  storage_message_id BIGINT NOT NULL,
  size_bytes BIGINT NOT NULL,
  etag TEXT NOT NULL,                     -- SHA256 hash as etag
  created_at TIMESTAMPTZ DEFAULT now(),
  UNIQUE(upload_id, part_number)
);

CREATE INDEX idx_multipart_parts_upload ON multipart_parts(upload_id, part_number);

2.5 Multipart Storage Strategy

Each part is stored as an independent file in the Telegram channel. The DB links them logically:

• UploadPart: forward part to Telegram via forwardToStorage → record in multipart_parts
• CompleteMultipartUpload: create a single files row with multipart_upload_id pointing to the parts. No new Telegram upload needed.
• GetObject for multipart object: query parts ordered by part_number, stream sequentially via Telegram CDN URLs. Client sees a single continuous download.
• AbortMultipartUpload: delete DB records only (Telegram orphan files are accepted as GC is not feasible)

---

3. S3 Protocol Layer

3.1 Full Endpoint Coverage (~20 endpoints)

Bucket Operations

Method	Path	Query Params	Handler Description
GET	/	—	List all buckets → <ListAllMyBucketsResult> XML
PUT	/{bucket}	—	Create bucket (reject if exists) → 200
HEAD	/{bucket}	—	Check bucket exists → 200 or 404
DELETE	/{bucket}	—	Delete empty bucket → 204

Object Operations

Method	Path	Query / Special	Handler Description
GET	/{bucket}/{key+}	—	GetObject: redirect to Telegram CDN (same as existing /f/:public_id)
HEAD	/{bucket}/{key+}	—	Return metadata headers (Content-Length, Content-Type, ETag, Last-Modified)
PUT	/{bucket}/{key+}	x-amz-copy-source?	Without copy header: upload multipart form → Telegram. With copy header: copy existing object in DB
DELETE	/{bucket}/{key+}	—	Soft-delete (is_deleted = true) → 204
POST	/{bucket}/{key+}	?tagging	Return 204 (no-op, tagging not implemented)

Object Listing

Method	Path	Query Params	Handler Description
GET	/{bucket}	—	ListObjects v1 → <ListBucketResult> XML
GET	/{bucket}	?list-type=2	ListObjectsV2 → <ListBucketResultV2> XML

Supports: prefix, delimiter, max-keys (default 1000), continuation-token (v2), marker (v1), encoding-type=url.

Batch Operations

Method	Path	Query Params	Handler Description
POST	/{bucket}	?delete	Parse XML body <Delete><Object><Key>...</Key></Object></Delete> → soft-delete each → <DeleteResult> XML

Multipart Upload

Method	Path	Query Params	Handler Description
POST	/{bucket}/{key+}	?uploads	Initiate: create multipart_uploads row → <InitiateMultipartUploadResult> XML with UploadId
PUT	/{bucket}/{key+}	?partNumber=N&uploadId=X	Upload part: stream to Telegram → record in multipart_parts → return ETag header
POST	/{bucket}/{key+}	?uploadId=X	Complete: parse <CompleteMultipartUpload><Part><PartNumber>N<ETag>...</Part></CompleteMultipartUpload> → create files row with multipart_upload_id → <CompleteMultipartUploadResult> XML
DELETE	/{bucket}/{key+}	?uploadId=X	Abort: delete records, update status → 204
GET	/{bucket}/{key+}	?uploadId=X	List parts → <ListPartsResult> XML

3.2 AWS Signature V4 (SigV4)

Authentication process for each S3 request:

1. Extract Authorization header
2. Parse credential scope (date/region/service)
3. Reconstruct CanonicalRequest
4. Compute expected signing key
5. Compare signatures
6. Match → proceed; Mismatch → 403 Forbidden with AWS XML error

Canonical Request construction:

<HTTPMethod>\n
<CanonicalURI>\n
<CanonicalQueryString>\n
<CanonicalHeaders>\n
<SignedHeaders>\n
<HashedPayload>

Implementation notes:

• Only verify signature; do not validate timestamp freshness (simplified, acceptable for single-server setup behind reverse proxy)
• Hard-code region = 'us-east-1' (irrelevant for functionality; S3 clients accept any region)
• Single set of S3 credentials (S3_ACCESS_KEY + S3_SECRET_KEY in env)
• Chunked transfer encoding (aws-chunked) — not supported initially; client must use standard PUT. Return 501 if detected.

3.3 XML Serialization

No external XML library. Template strings for responses:

// Example: ListBuckets response
const listBucketsXml = (buckets: Bucket[]) => `<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Buckets>
    ${buckets.map(b => `<Bucket>
      <Name>${escapeXml(b.name)}</Name>
      <CreationDate>${b.createdAt.toISOString()}</CreationDate>
    </Bucket>`).join('')}
  </Buckets>
</ListAllMyBucketsResult>`;

XML parsing (only for DeleteObjects body): use simple string matching / regex on <Key>...</Key> tags. The DeleteObjects XML is simple and well-structured enough for this without a parser library.

3.4 Error Responses

All S3 errors return XML with HTTP status:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
  <Resource>/bucket/key</Resource>
  <RequestId>...</RequestId>
</Error>

Common error codes: NoSuchBucket, NoSuchKey, BucketAlreadyExists, BucketNotEmpty, InvalidPartOrder, NoSuchUpload, SignatureDoesNotMatch, AccessDenied, InternalError.

3.5 Presigned URL Support

GET /{bucket}/{key+}?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature=...

For presigned GET URLs:

• Verify signature (much simpler — no body hash required, querystring-based)
• If valid → return redirect to Telegram CDN (same as GetObject)
• If expired → 403

Only presigned GET is essential; presigned PUT is optional for v1.

---

4. Web File Manager UI

4.1 Served At

Route '/' → home.html (single HTML file with embedded CSS + JS)

4.2 Layout (responsive)

┌─────────────────────────────────────────┐
│ ☰ TeleUploader    ◉ my-bucket ▼ [+]     │  ← Top bar
├─────────────────────────────────────────┤
│ ─────────────────────────────────────── │
│ 🗂  images/            Jul 06   2 items  │
│ 🗂  documents/         Jul 05   5 items  │
│ 📄 logo.png    2.4 MB Jul 06  [⋮ ▼]     │  ← Actions dropdown
│ 📄 report.pdf  1.2 MB Jul 05  [⋮ ▼]     │
│ 📄 photo.jpg   3.1 MB Jul 04  [⋮ ▼]     │
│ ─────────────────────────────────────── │
│                              ↑ Load more │
└─────────────────────────────────────────┘
│ Drag & drop upload area (footer)        │
└─────────────────────────────────────────┘

4.3 Features

Feature	Implementation
Bucket selector	Dropdown, fetches bucket list from /api/v1/buckets
Create bucket	Prompt for name → POST /api/v1/buckets
Delete bucket	Confirm → DELETE /api/v1/buckets/{name}
Object listing	GET /api/v1/buckets/{name}/objects?prefix=X&delimiter=/
Folder navigation	Breadcrumb from prefix, click to drill down
Upload	Drag-drop or click → POST /api/v1/buckets/{name}/upload
Upload progress	XMLHttpRequest.upload.onprogress
Download	Direct download via GET /api/v1/buckets/{name}/download/{key}
Copy link	Copy full S3 URL to clipboard
Delete	Confirm → soft delete
Search	Filter by key prefix (debounced)
Credentials info	Show S3_ACCESS_KEY + S3_SECRET_KEY (from env) in a modal
Dark/light mode	CSS variables, @media prefers-color-scheme

4.4 Web API v1 (JSON)

These endpoints enable the UI without XML:

Method	Path	Response
GET	/api/v1/buckets	{buckets: [{id, name, createdAt, objectCount}]}
POST	/api/v1/buckets	{id, name}
DELETE	/api/v1/buckets/{name}	{success: true}
GET	/api/v1/buckets/{name}/objects?prefix=&delimiter=/&continuationToken=	{objects, prefixes, isTruncated}
POST	/api/v1/buckets/{name}/upload	Multipart form → {key, size, etag}
DELETE	/api/v1/buckets/{name}/{key+}	{success: true}
POST	/api/v1/buckets/{name}/copy	{sourceKey, destKey}

4.5 Download via Web UI

For the web UI, download doesn't use S3 headers. Instead:

• GET /api/v1/buckets/{name}/download/{key+} → redirects to Telegram CDN, similar to /f/:public_id
• This bypasses S3 auth (which the browser can't do with SigV4)

---

5. Existing Routes — Unchanged

All existing functionality remains intact:

• POST /api/upload — multipart + JSON file upload
• GET /f/:public_id — file redirect to Telegram CDN
• GET /file/:public_id/info — file metadata
• GET /health — database health
• GET /docs + /swagger.json — API docs

The existing upload flow does not interact with S3 bucket/keys. This is intentional: the JSON API remains for simple programmatic upload without S3 complexity.

---

6. Security Model

Layer	Mechanism
S3 API	SigV4 signature verification. Single credential pair from env
Web UI	No auth (internal tool). Relies on network-level security / reverse proxy
JSON API (existing)	Rate-limited only (same as now). No additional auth
Rate limiting	Applied to S3 operations per IP

---

7. Implementation Order

The following build sequence minimizes blocked dependencies:

Phase	Tasks	Depends On
1. DB schema	Create buckets, multipart_uploads, multipart_parts tables. Migrate existing DB	Nothing
2. DB layer	CRUD functions for buckets, multipart, files extended	Phase 1
3. S3 Auth + XML	SigV4 verification, XML builder templates	Nothing (parallel with 1)
4. S3 Bucket Ops	ListBuckets, CreateBucket, HeadBucket, DeleteBucket	Phase 1, 3
5. S3 Object Ops	PutObject, GetObject, HeadObject, DeleteObject, CopyObject, DeleteObjects	Phase 2, 3, 4
6. S3 Listing	ListObjects, ListObjectsV2	Phase 2, 3, 4
7. S3 Multipart	CreateMultipartUpload, UploadPart, Complete, Abort, ListParts	Phase 2, 3, 4
8. S3 Dispatcher	Wire up catch-all route with auth detection	Phase 4-7
9. JSON Web API	All /api/v1/* endpoints	Phase 2, 4
10. Web UI	home.html with bucket browser, upload, delete, search	Phase 9
11. Tests	S3 auth test, bucket operations test, object operations test, multipart test	Phase 4-8

---

8. Configuration (New Env Vars)

# S3-compatible API credentials
S3_ACCESS_KEY=teleuploader-admin       # Access key for SigV4 auth
S3_SECRET_KEY=your-secret-key-here     # Secret key for SigV4 auth
S3_DEFAULT_REGION=us-east-1            # S3 region (cosmetic, affects signature scope)

# Existing vars unchanged:
# BOT_TOKEN, ADDITIONAL_BOT_TOKENS, STORAGE_CHANNEL_ID, BASE_URL, ...

---

9. Testing Strategy

Test Focus	Scope
SigV4 auth	Verify signature verification, reject invalid signatures
S3 Bucket operations	Create, list, head, delete buckets
S3 Object operations	Put, get, head, delete objects (via Telegram)
S3 ListObjects	Pagination, prefix, delimiter, continuation token
S3 Multipart	Create, upload parts, complete, abort
S3 DeleteObjects	Batch delete XML body parsing
S3 Error responses	Correct XML error format per status code
Web API	JSON endpoints return correct data
Non-S3 routes	Existing routes still work (regression)

---

10. Open Questions / Future

1. Presigned URL support — GET presigned URLs in v1; PUT presigned for v2
2. CORS headers — if web UI served from different origin than S3 API calls
3. Versioning — not supported (single version per key)
4. ACL / Bucket policies — not supported (single user model)
5. Lifecycle rules — not supported
6. Website hosting — not supported
7. Default bucket — consider creating a default bucket on first start for convenience
8. Upload via S3 API directly to web UI — web UI could also call S3 API directly for maximum compatibility demo
9. S3-compatible client list — tested clients: aws-cli, rclone, s3cmd, MinIO Client, Cyberduck