2956 lines
103 KiB
Markdown
2956 lines
103 KiB
Markdown
# S3-Compatible TeleUploader Implementation Plan
|
|
|
|
> ⚠️ **LEGACY** — Dokumen historis (2026-07-06). Port & infrastruktur sudah berubah:
|
|
> produksi kini berjalan di port `4000` (Nix + systemd + Caddy, domain
|
|
> `upload.asepharyana.my.id`), database via PgBouncer pool `100.121.180.82:6432`
|
|
> (bukan 5432/localhost). Contoh kode di bawah memakai `localhost:4000` untuk dev.
|
|
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Transform TeleUploader into an S3-compatible storage server (Telegram-backed) with a web file manager UI.
|
|
|
|
**Architecture:** An S3 protocol dispatcher at the catch-all route (`$`) intercepts SigV4-authenticated requests and routes them to ~20 S3-compatible endpoints. A JSON v1 API layer wraps the same operations for use by a single-page web file manager served at `/`. Storage remains Telegram (single channel); S3 buckets are virtual entities in PostgreSQL.
|
|
|
|
**Tech Stack:** Bun, TypeScript, PostgreSQL (Drizzle ORM), raw AWS SigV4 (no library), zero external XML deps
|
|
|
|
## Global Constraints
|
|
|
|
- All new code follows existing codebase style (Bun, TypeScript, Drizzle ORM)
|
|
- Existing routes (`/api/upload`, `/f/:public_id`, `/file/:public_id/info`, `/health`, `/docs`) must remain unchanged and functional
|
|
- No external libraries for S3 protocol, auth, or XML handling
|
|
- All S3 error responses must use correct XML format with AWS error codes
|
|
- Only one S3 credential pair (`S3_ACCESS_KEY` + `S3_SECRET_KEY`) in env config
|
|
- Hard-code S3 region as `us-east-1`
|
|
- Chunked transfer encoding (`aws-chunked`) not supported — return 501
|
|
- Presigned GET URLs supported; presigned PUT is optional for v1
|
|
- Tests use Bun's built-in test runner and mock patterns (`mock.module()`)
|
|
- Files in `schema.sql` are raw SQL (not Drizzle migrations); new tables go there
|
|
- All `/api/v1/*` routes are unprotected (no SigV4); rely on network-level security
|
|
- Avoid ORM features that don't work with raw `schema.sql` and `postgres` driver — use raw SQL queries or Drizzle's `sql` template tag for complex queries
|
|
|
|
---
|
|
|
|
### File Structure
|
|
|
|
#### New files to create:
|
|
| File | Responsibility |
|
|
|------|----------------|
|
|
| `src/utils/s3/auth.ts` | AWS SigV4 signature verification, signing key derivation, canonical request construction |
|
|
| `src/utils/s3/xml.ts` | S3 XML response builders (all 20+ endpoint templates), XML parser for DeleteObjects body |
|
|
| `src/db/buckets.ts` | CRUD for `buckets` table: create, list, findByName, delete |
|
|
| `src/db/multipart.ts` | CRUD for `multipart_uploads` and `multipart_parts` tables: create, list parts, complete, abort, insert part |
|
|
| `src/db/files-ext.ts` | Extended file queries: findByBucketAndKey, listByPrefix, findActiveByBucket, softDelete, copyObject, listByBucketForUICount |
|
|
| `src/routes/s3.ts` | S3 protocol dispatcher and all ~20 S3 operation handlers |
|
|
| `src/routes/web-api.ts` | JSON v1 API endpoints for web UI (buckets, objects, upload, download, copy, delete) |
|
|
| `src/routes/home.ts` | Serves the file manager HTML at `/` |
|
|
| `src/home.html` | Web file manager SPA with embedded CSS/JS |
|
|
| `test/s3-auth.test.ts` | Tests for SigV4 signature verification |
|
|
| `test/s3-operations.test.ts` | Tests for S3 bucket/object/multipart operations |
|
|
| `test/web-api.test.ts` | Tests for JSON v1 API endpoints |
|
|
|
|
#### Existing files to modify:
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `schema.sql` | Add `buckets`, `multipart_uploads`, `multipart_parts` tables; add columns to `files` |
|
|
| `src/env.ts` | Add `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_DEFAULT_REGION` config fields |
|
|
| `src/index.ts` | Import and register home, web-api, and S3 catch-all routes |
|
|
| `src/db/files.ts` | No changes needed (existing queries remain); new queries go in `files-ext.ts` |
|
|
| `.env.example` | Add commented S3 env vars |
|
|
| `tsconfig.json` | No changes needed |
|
|
|
|
---
|
|
|
|
### Task 1: Environment Config + DB Schema
|
|
|
|
**Files:**
|
|
- Modify: `src/env.ts`
|
|
- Modify: `schema.sql`
|
|
- Modify: `.env.example`
|
|
|
|
**Interfaces:**
|
|
- Consumes: existing `src/env.ts` pattern
|
|
- Produces: `config.s3AccessKey`, `config.s3SecretKey`, `config.s3DefaultRegion` env fields; new DB tables and columns
|
|
|
|
- [ ] **Step 1: Add S3 env vars to `src/env.ts`**
|
|
|
|
Add to the `AppConfig` interface:
|
|
```typescript
|
|
interface AppConfig {
|
|
// ... existing fields ...
|
|
s3AccessKey: string;
|
|
s3SecretKey: string;
|
|
s3DefaultRegion: string;
|
|
}
|
|
```
|
|
|
|
Add to the config object after the `maxRequestBodyBytes` line:
|
|
```typescript
|
|
s3AccessKey: process.env.S3_ACCESS_KEY || 'teleuploader-admin',
|
|
s3SecretKey: process.env.S3_SECRET_KEY || '',
|
|
s3DefaultRegion: process.env.S3_DEFAULT_REGION || 'us-east-1',
|
|
```
|
|
|
|
Add S3 vars to the logger.info config block.
|
|
|
|
- [ ] **Step 2: Update `schema.sql` — add new tables and columns**
|
|
|
|
Append after the existing files table:
|
|
|
|
```sql
|
|
-- S3-compatible buckets
|
|
CREATE TABLE IF NOT EXISTS buckets (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
name VARCHAR(63) UNIQUE NOT NULL,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
-- Extend files table for S3
|
|
ALTER TABLE files ADD COLUMN IF NOT EXISTS bucket_id UUID REFERENCES buckets(id);
|
|
ALTER TABLE files ADD COLUMN IF NOT EXISTS s3_key TEXT;
|
|
ALTER TABLE files ADD COLUMN IF NOT EXISTS storage_backend VARCHAR DEFAULT 'telegram';
|
|
ALTER TABLE files ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN DEFAULT false;
|
|
ALTER TABLE files ADD COLUMN IF NOT EXISTS multipart_upload_id TEXT;
|
|
|
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_files_bucket_key ON files(bucket_id, s3_key) WHERE is_deleted = false;
|
|
CREATE INDEX IF NOT EXISTS idx_files_bucket_prefix ON files(bucket_id, s3_key text_pattern_ops);
|
|
CREATE INDEX IF NOT EXISTS idx_files_s3_key ON files(s3_key);
|
|
CREATE INDEX IF NOT EXISTS idx_files_bucket_id ON files(bucket_id);
|
|
|
|
-- Multipart upload tracking
|
|
CREATE TABLE IF NOT EXISTS multipart_uploads (
|
|
upload_id VARCHAR PRIMARY KEY,
|
|
bucket_id UUID NOT NULL REFERENCES buckets(id),
|
|
s3_key TEXT NOT NULL,
|
|
initiated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
status VARCHAR DEFAULT 'in_progress',
|
|
initiated_by TEXT
|
|
);
|
|
|
|
CREATE TABLE IF NOT EXISTS multipart_parts (
|
|
id SERIAL PRIMARY KEY,
|
|
upload_id VARCHAR NOT NULL REFERENCES multipart_uploads(upload_id) ON DELETE CASCADE,
|
|
part_number INT NOT NULL,
|
|
telegram_file_id VARCHAR NOT NULL,
|
|
telegram_file_unique_id VARCHAR NOT NULL,
|
|
storage_message_id BIGINT NOT NULL,
|
|
size_bytes BIGINT NOT NULL,
|
|
etag VARCHAR NOT NULL,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
UNIQUE(upload_id, part_number)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_multipart_parts_upload ON multipart_parts(upload_id, part_number);
|
|
CREATE INDEX IF NOT EXISTS idx_multipart_uploads_status ON multipart_uploads(status);
|
|
```
|
|
|
|
- [ ] **Step 3: Update `.env.example`**
|
|
|
|
Append to .env.example:
|
|
```env
|
|
# S3-compatible API credentials
|
|
# S3_ACCESS_KEY=teleuploader-admin
|
|
# S3_SECRET_KEY=your-secret-key-here
|
|
# S3_DEFAULT_REGION=us-east-1
|
|
```
|
|
|
|
- [ ] **Step 4: Run migration to verify**
|
|
|
|
```bash
|
|
cd /mnt/code/TeleUploader && bun run db:migrate
|
|
```
|
|
|
|
Expected: `Database migration completed` logged. No errors.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/env.ts schema.sql .env.example
|
|
git commit -m "feat: add S3 env config, bucket and multipart DB schema"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Database CRUD Layer
|
|
|
|
**Files:**
|
|
- Create: `src/db/buckets.ts`
|
|
- Create: `src/db/multipart.ts`
|
|
- Create: `src/db/files-ext.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `src/db/index.ts` (existing `db` and `files`), `config` from `src/env.ts`
|
|
- Produces: Bucket CRUD functions, Multipart CRUD functions, Extended file queries by bucket+key
|
|
|
|
- [ ] **Step 1: Create `src/db/buckets.ts`**
|
|
|
|
```typescript
|
|
import { sql } from 'drizzle-orm';
|
|
import { db } from './index';
|
|
import logger from '../utils/logger';
|
|
|
|
export interface Bucket {
|
|
id: string;
|
|
name: string;
|
|
createdAt: Date;
|
|
updatedAt: Date;
|
|
}
|
|
|
|
export const createBucket = async (name: string): Promise<Bucket> => {
|
|
const result = await db.execute(
|
|
sql`INSERT INTO buckets (name) VALUES (${name}) RETURNING id, name, created_at, updated_at`,
|
|
);
|
|
const row = result.rows[0] as Record<string, unknown>;
|
|
return {
|
|
id: row.id as string,
|
|
name: row.name as string,
|
|
createdAt: new Date(row.created_at as string),
|
|
updatedAt: new Date(row.updated_at as string),
|
|
};
|
|
};
|
|
|
|
export const findBucketByName = async (name: string): Promise<Bucket | null> => {
|
|
const result = await db.execute(sql`SELECT id, name, created_at, updated_at FROM buckets WHERE name = ${name}`);
|
|
if (result.rows.length === 0) return null;
|
|
const row = result.rows[0] as Record<string, unknown>;
|
|
return {
|
|
id: row.id as string,
|
|
name: row.name as string,
|
|
createdAt: new Date(row.created_at as string),
|
|
updatedAt: new Date(row.updated_at as string),
|
|
};
|
|
};
|
|
|
|
export const listBuckets = async (): Promise<Bucket[]> => {
|
|
const result = await db.execute(sql`SELECT id, name, created_at, updated_at FROM buckets ORDER BY name`);
|
|
return result.rows.map((row) => {
|
|
const r = row as Record<string, unknown>;
|
|
return {
|
|
id: r.id as string,
|
|
name: r.name as string,
|
|
createdAt: new Date(r.created_at as string),
|
|
updatedAt: new Date(r.updated_at as string),
|
|
};
|
|
});
|
|
};
|
|
|
|
export const deleteBucket = async (name: string): Promise<boolean> => {
|
|
const result = await db.execute(sql`DELETE FROM buckets WHERE name = ${name}`);
|
|
return (result as unknown as { rowCount: number }).rowCount > 0;
|
|
};
|
|
|
|
export const bucketExists = async (name: string): Promise<boolean> => {
|
|
const result = await db.execute(sql`SELECT 1 FROM buckets WHERE name = ${name}`);
|
|
return result.rows.length > 0;
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 2: Create `src/db/multipart.ts`**
|
|
|
|
```typescript
|
|
import { sql } from 'drizzle-orm';
|
|
import { db } from './index';
|
|
import logger from '../utils/logger';
|
|
import { nanoid } from 'nanoid';
|
|
|
|
export interface MultipartUpload {
|
|
uploadId: string;
|
|
bucketId: string;
|
|
s3Key: string;
|
|
initiatedAt: Date;
|
|
status: string;
|
|
initiatedBy: string;
|
|
}
|
|
|
|
export interface MultipartPart {
|
|
id: number;
|
|
uploadId: string;
|
|
partNumber: number;
|
|
telegramFileId: string;
|
|
telegramFileUniqueId: string;
|
|
storageMessageId: number;
|
|
sizeBytes: number;
|
|
etag: string;
|
|
createdAt: Date;
|
|
}
|
|
|
|
export const createMultipartUpload = async (
|
|
bucketId: string,
|
|
s3Key: string,
|
|
initiatedBy: string,
|
|
): Promise<string> => {
|
|
const uploadId = nanoid(32);
|
|
await db.execute(
|
|
sql`INSERT INTO multipart_uploads (upload_id, bucket_id, s3_key, initiated_by) VALUES (${uploadId}, ${bucketId}, ${s3Key}, ${initiatedBy})`,
|
|
);
|
|
return uploadId;
|
|
};
|
|
|
|
export const findMultipartUpload = async (uploadId: string): Promise<MultipartUpload | null> => {
|
|
const result = await db.execute(
|
|
sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status FROM multipart_uploads WHERE upload_id = ${uploadId} AND status = 'in_progress'`,
|
|
);
|
|
if (result.rows.length === 0) return null;
|
|
const r = result.rows[0] as Record<string, unknown>;
|
|
return {
|
|
uploadId: r.upload_id as string,
|
|
bucketId: r.bucket_id as string,
|
|
s3Key: r.s3_key as string,
|
|
initiatedAt: new Date(r.initiated_at as string),
|
|
status: r.status as string,
|
|
initiatedBy: '',
|
|
};
|
|
};
|
|
|
|
export const completeMultipartUpload = async (uploadId: string): Promise<void> => {
|
|
await db.execute(sql`UPDATE multipart_uploads SET status = 'completed' WHERE upload_id = ${uploadId}`);
|
|
};
|
|
|
|
export const abortMultipartUpload = async (uploadId: string): Promise<void> => {
|
|
await db.execute(sql`UPDATE multipart_uploads SET status = 'aborted' WHERE upload_id = ${uploadId}`);
|
|
// Parts are cascade-deleted by FK
|
|
};
|
|
|
|
export const insertMultipartPart = async (part: Omit<MultipartPart, 'id' | 'createdAt'>): Promise<void> => {
|
|
await db.execute(
|
|
sql`INSERT INTO multipart_parts (upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag)
|
|
VALUES (${part.uploadId}, ${part.partNumber}, ${part.telegramFileId}, ${part.telegramFileUniqueId}, ${part.storageMessageId}, ${part.sizeBytes}, ${part.etag})`,
|
|
);
|
|
};
|
|
|
|
export const listMultipartParts = async (uploadId: string): Promise<MultipartPart[]> => {
|
|
const result = await db.execute(
|
|
sql`SELECT id, upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag, created_at
|
|
FROM multipart_parts WHERE upload_id = ${uploadId} ORDER BY part_number`,
|
|
);
|
|
return result.rows.map((row) => {
|
|
const r = row as Record<string, unknown>;
|
|
return {
|
|
id: r.id as number,
|
|
uploadId: r.upload_id as string,
|
|
partNumber: r.part_number as number,
|
|
telegramFileId: r.telegram_file_id as string,
|
|
telegramFileUniqueId: r.telegram_file_unique_id as string,
|
|
storageMessageId: r.storage_message_id as number,
|
|
sizeBytes: r.size_bytes as number,
|
|
etag: r.etag as string,
|
|
createdAt: new Date(r.created_at as string),
|
|
};
|
|
});
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 3: Create `src/db/files-ext.ts`**
|
|
|
|
```typescript
|
|
import { eq, and, isNull, sql } from 'drizzle-orm';
|
|
import { db, files as fileSchema } from './index';
|
|
import type { File } from './schema';
|
|
import logger from '../utils/logger';
|
|
|
|
export interface S3FileRecord extends File {
|
|
bucketId: string;
|
|
s3Key: string;
|
|
}
|
|
|
|
export const findFileByBucketAndKey = async (bucketId: string, s3Key: string): Promise<File | null> => {
|
|
const result = await db
|
|
.select()
|
|
.from(fileSchema)
|
|
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.s3Key, s3Key), eq(fileSchema.isDeleted, false)))
|
|
.limit(1);
|
|
return result[0] || null;
|
|
};
|
|
|
|
export interface ListObjectsRow {
|
|
key: string;
|
|
fileName: string;
|
|
sizeBytes: number;
|
|
mimeType: string;
|
|
etag: string;
|
|
lastModified: Date;
|
|
isDeleted: boolean;
|
|
}
|
|
|
|
export const listObjectsByPrefix = async (
|
|
bucketId: string,
|
|
prefix: string,
|
|
delimiter: string | null,
|
|
maxKeys: number,
|
|
startAfter: string | null,
|
|
): Promise<{ objects: S3FileRecord[]; prefixes: string[] }> => {
|
|
const keys = [];
|
|
|
|
// Base query: active files in this bucket with key starting with prefix
|
|
const conditions = [sql`bucket_id = ${bucketId}::uuid`, sql`is_deleted = false`, sql`s3_key LIKE ${prefix + '%'}`];
|
|
const orderClause = sql`ORDER BY s3_key`;
|
|
|
|
if (startAfter) {
|
|
conditions.push(sql`s3_key > ${startAfter}`);
|
|
}
|
|
|
|
const whereClause = conditions.map((c) => c?.text?.replace(/\$(\d+)/g, ''));
|
|
|
|
const result = await db
|
|
.select()
|
|
.from(fileSchema)
|
|
.where(and(...conditions.map((c) => sql`${c}`)) as unknown as ReturnType<typeof and>)
|
|
.orderBy(fileSchema.s3Key as unknown as 'asc' | 'desc')
|
|
.limit(maxKeys + 1);
|
|
|
|
if (delimiter === '/') {
|
|
const prefixSet = new Set<string>();
|
|
const objects: S3FileRecord[] = [];
|
|
|
|
for (const file of result) {
|
|
const relativeKey = file.s3Key!.substring(prefix.length);
|
|
const slashIndex = relativeKey.indexOf('/');
|
|
if (slashIndex >= 0) {
|
|
// It's under a subfolder — extract the folder prefix
|
|
const folderPrefix = prefix + relativeKey.substring(0, slashIndex + 1);
|
|
if (folderPrefix !== prefix) {
|
|
prefixSet.add(folderPrefix);
|
|
}
|
|
} else {
|
|
// It's a direct child object
|
|
objects.push(file as unknown as S3FileRecord);
|
|
}
|
|
}
|
|
|
|
return {
|
|
objects: objects.slice(0, maxKeys),
|
|
prefixes: Array.from(prefixSet).sort(),
|
|
};
|
|
}
|
|
|
|
return {
|
|
objects: result as unknown as S3FileRecord[],
|
|
prefixes: [],
|
|
};
|
|
};
|
|
|
|
export const softDeleteFile = async (bucketId: string, s3Key: string): Promise<boolean> => {
|
|
const result = await db
|
|
.update(fileSchema)
|
|
.set({ isDeleted: true })
|
|
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.s3Key, s3Key)))
|
|
.returning();
|
|
return result.length > 0;
|
|
};
|
|
|
|
export const softDeleteFilesBatch = async (bucketId: string, keys: string[]): Promise<number> => {
|
|
let deleted = 0;
|
|
for (const key of keys) {
|
|
const ok = await softDeleteFile(bucketId, key);
|
|
if (ok) deleted++;
|
|
}
|
|
return deleted;
|
|
};
|
|
|
|
export const countBucketObjects = async (bucketId: string): Promise<number> => {
|
|
const result = await db
|
|
.select({ count: sql<number>`count(*)` })
|
|
.from(fileSchema)
|
|
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.isDeleted, false)));
|
|
return Number(result[0]?.count || 0);
|
|
};
|
|
|
|
export const findOrphanFilesByBucket = async (bucketId: string): Promise<File[]> => {
|
|
return await db
|
|
.select()
|
|
.from(fileSchema)
|
|
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.isDeleted, true)))
|
|
.limit(100);
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add src/db/buckets.ts src/db/multipart.ts src/db/files-ext.ts
|
|
git commit -m "feat: add DB CRUD layer for buckets, multipart, and S3 file extensions"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: S3 Auth (SigV4) + XML Utilities
|
|
|
|
**Files:**
|
|
- Create: `src/utils/s3/auth.ts`
|
|
- Create: `src/utils/s3/xml.ts`
|
|
|
|
**Interfaces:**
|
|
- Produces: `verifySignature(req, s3AccessKey, s3SecretKey, region, bucket, key) → { isValid, credential }`
|
|
- Produces: `XML builder functions` for all S3 responses, `parseDeleteObjectsXml(body) → string[]`
|
|
|
|
- [ ] **Step 1: Create `src/utils/s3/auth.ts`**
|
|
|
|
The `crypto` module in Bun uses Web Crypto API. For HMAC-SHA256, use `crypto.subtle`:
|
|
|
|
```typescript
|
|
import { config } from '../../env';
|
|
|
|
export interface SigV4Result {
|
|
isValid: boolean;
|
|
credential: {
|
|
accessKey: string;
|
|
date: string;
|
|
region: string;
|
|
service: string;
|
|
} | null;
|
|
errorCode?: string;
|
|
}
|
|
|
|
const SERVICE = 's3';
|
|
const TERMINATION = 'aws4_request';
|
|
|
|
// Create SHA-256 hash
|
|
const sha256 = async (data: string | BufferSource): Promise<string> => {
|
|
const encoder = new TextEncoder();
|
|
const dataBuffer = typeof data === 'string' ? encoder.encode(data) : data;
|
|
const hashBuffer = await crypto.subtle.digest('SHA-256', dataBuffer);
|
|
const hashArray = Array.from(new Uint8Array(hashBuffer));
|
|
return hashArray.map((b) => b.toString(16).padStart(2, '0')).join('');
|
|
};
|
|
|
|
// HMAC-SHA256
|
|
const hmacSha256 = async (key: BufferSource, message: string): Promise<ArrayBuffer> => {
|
|
const cryptoKey = await crypto.subtle.importKey(
|
|
'raw',
|
|
key,
|
|
{ name: 'HMAC', hash: 'SHA-256' },
|
|
false,
|
|
['sign'],
|
|
);
|
|
const encoder = new TextEncoder();
|
|
return await crypto.subtle.sign('HMAC', cryptoKey, encoder.encode(message));
|
|
};
|
|
|
|
// Derive signing key
|
|
const getSigningKey = async (secretKey: string, dateStamp: string, region: string): Promise<ArrayBuffer> => {
|
|
const encoder = new TextEncoder();
|
|
let key = await hmacSha256(encoder.encode(`AWS4${secretKey}`), dateStamp);
|
|
key = await hmacSha256(key, region);
|
|
key = await hmacSha256(key, SERVICE);
|
|
return await hmacSha256(key, TERMINATION);
|
|
};
|
|
|
|
// Hex-encode HMAC result
|
|
const hmacHex = async (key: BufferSource, message: string): Promise<string> => {
|
|
const result = await hmacSha256(key, message);
|
|
const hashArray = Array.from(new Uint8Array(result));
|
|
return hashArray.map((b) => b.toString(16).padStart(2, '0')).join('');
|
|
};
|
|
|
|
// Parse AWS4-HMAC-SHA256 Authorization header
|
|
const parseAuthorizationHeader = (authHeader: string) => {
|
|
// "AWS4-HMAC-SHA256 Credential=AKID/20260706/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=..."
|
|
const credentialMatch = authHeader.match(/Credential=([^,]+)/);
|
|
const signedHeadersMatch = authHeader.match(/SignedHeaders=([^,]+)/);
|
|
const signatureMatch = authHeader.match(/Signature=([^,]+)/);
|
|
|
|
if (!credentialMatch || !signedHeadersMatch || !signatureMatch) return null;
|
|
|
|
const credentialParts = credentialMatch[1].split('/');
|
|
if (credentialParts.length !== 5) return null;
|
|
|
|
return {
|
|
accessKey: credentialParts[0],
|
|
date: credentialParts[1],
|
|
region: credentialParts[2],
|
|
service: credentialParts[3],
|
|
termination: credentialParts[4],
|
|
signedHeaders: signedHeadersMatch[1],
|
|
signature: signatureMatch[1],
|
|
};
|
|
};
|
|
|
|
// Build canonical request
|
|
const buildCanonicalRequest = (
|
|
method: string,
|
|
canonicalUri: string,
|
|
canonicalQueryString: string,
|
|
signedHeaders: string,
|
|
headers: Record<string, string>,
|
|
hashedPayload: string,
|
|
): string => {
|
|
const canonicalHeaders = signedHeaders
|
|
.split(';')
|
|
.map((h) => {
|
|
const value = headers[h.toLowerCase()] || '';
|
|
return `${h.toLowerCase()}:${value.trim()}\n`;
|
|
})
|
|
.join('');
|
|
|
|
return `${method}\n${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeaders}\n${hashedPayload}`;
|
|
};
|
|
|
|
// Normalize URI (S3 requires URI-encoded paths but decoded for canonical request)
|
|
const normalizeUri = (uri: string): string => {
|
|
if (!uri || uri === '') return '/';
|
|
return uri;
|
|
};
|
|
|
|
// Build canonical query string from URLSearchParams
|
|
const buildCanonicalQueryString = (searchParams: URLSearchParams): string => {
|
|
const params: string[] = [];
|
|
// Sort by key, then by value
|
|
const keys = Array.from(searchParams.keys()).sort();
|
|
for (const key of keys) {
|
|
const values = searchParams.getAll(key).sort();
|
|
for (const value of values) {
|
|
params.push(`${encodeURIComponent(key)}=${encodeURIComponent(value)}`);
|
|
}
|
|
}
|
|
return params.join('&');
|
|
};
|
|
|
|
// Get hashed payload from x-amz-content-sha256 header or body
|
|
const getHashedPayload = async (body: string | null, contentSha256: string | null): Promise<string> => {
|
|
if (contentSha256) return contentSha256;
|
|
if (!body || body.length === 0) return await sha256('');
|
|
return await sha256(body);
|
|
};
|
|
|
|
export const verifySignature = async (
|
|
method: string,
|
|
url: string,
|
|
headers: Record<string, string>,
|
|
body: string | null,
|
|
s3AccessKey: string,
|
|
s3SecretKey: string,
|
|
region: string,
|
|
): Promise<SigV4Result> => {
|
|
const authHeader = headers['authorization'];
|
|
if (!authHeader || !authHeader.startsWith('AWS4-HMAC-SHA256')) {
|
|
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
|
|
}
|
|
|
|
const parsed = parseAuthorizationHeader(authHeader);
|
|
if (!parsed) {
|
|
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
|
|
}
|
|
|
|
// Reject if access key doesn't match
|
|
if (parsed.accessKey !== s3AccessKey) {
|
|
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
|
|
}
|
|
|
|
const parsedUrl = new URL(url, 'http://localhost');
|
|
const canonicalUri = normalizeUri(parsedUrl.pathname);
|
|
const canonicalQueryString = buildCanonicalQueryString(parsedUrl.searchParams);
|
|
|
|
const contentSha256 = headers['x-amz-content-sha256'] || null;
|
|
const hashedPayload = await getHashedPayload(body, contentSha256);
|
|
|
|
const canonicalRequest = buildCanonicalRequest(
|
|
method,
|
|
canonicalUri,
|
|
canonicalQueryString,
|
|
parsed.signedHeaders,
|
|
headers,
|
|
hashedPayload,
|
|
);
|
|
|
|
const hashedCanonicalRequest = await sha256(canonicalRequest);
|
|
|
|
const amzDate = headers['x-amz-date'] || '';
|
|
const dateStamp = parsed.date; // YYYYMMDD from credential
|
|
const credentialScope = `${dateStamp}/${parsed.region}/${parsed.service}/${parsed.termination}`;
|
|
|
|
const stringToSign = `AWS4-HMAC-SHA256\n${amzDate}\n${credentialScope}\n${hashedCanonicalRequest}`;
|
|
|
|
const signingKey = await getSigningKey(s3SecretKey, dateStamp, region);
|
|
const expectedSignature = await hmacHex(signingKey, stringToSign);
|
|
|
|
if (expectedSignature !== parsed.signature) {
|
|
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
|
|
}
|
|
|
|
return {
|
|
isValid: true,
|
|
credential: {
|
|
accessKey: parsed.accessKey,
|
|
date: parsed.date,
|
|
region: parsed.region,
|
|
service: parsed.service,
|
|
},
|
|
};
|
|
};
|
|
|
|
// Simplified verify for presigned URLs
|
|
export const verifyPresignedUrl = async (
|
|
url: string,
|
|
s3AccessKey: string,
|
|
s3SecretKey: string,
|
|
region: string,
|
|
): Promise<SigV4Result> => {
|
|
const parsedUrl = new URL(url);
|
|
const queryParams = Object.fromEntries(parsedUrl.searchParams.entries());
|
|
|
|
const algorithm = queryParams['X-Amz-Algorithm'];
|
|
const credential = queryParams['X-Amz-Credential'];
|
|
const signedHeaders = queryParams['X-Amz-SignedHeaders'];
|
|
const signature = queryParams['X-Amz-Signature'];
|
|
const expires = parseInt(queryParams['X-Amz-Expires'] || '0', 10);
|
|
const amzDate = queryParams['X-Amz-Date'];
|
|
|
|
if (!algorithm || algorithm !== 'AWS4-HMAC-SHA256' || !credential || !signature || !expires || !amzDate) {
|
|
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
|
|
}
|
|
|
|
// Check expiration
|
|
const dateObj = new Date(
|
|
parseInt(amzDate.substring(0, 4), 10),
|
|
parseInt(amzDate.substring(4, 6), 10) - 1,
|
|
parseInt(amzDate.substring(6, 8), 10),
|
|
parseInt(amzDate.substring(9, 11), 10),
|
|
parseInt(amzDate.substring(11, 13), 10),
|
|
parseInt(amzDate.substring(13, 15), 10),
|
|
);
|
|
const expiresMs = expires * 1000;
|
|
if (Date.now() > dateObj.getTime() + expiresMs) {
|
|
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
|
|
}
|
|
|
|
const credParts = credential.split('/');
|
|
const dateStamp = credParts[1] || amzDate.substring(0, 8);
|
|
|
|
// Build canonical request for presigned URL (no body hash — unsigned-payload)
|
|
const canonicalUri = normalizeUri(parsedUrl.pathname);
|
|
|
|
// Sort query params (excluding signature)
|
|
const sortedParams = new URLSearchParams();
|
|
const paramKeys = Object.keys(queryParams).sort();
|
|
for (const key of paramKeys) {
|
|
if (key !== 'X-Amz-Signature') {
|
|
sortedParams.append(key, queryParams[key]);
|
|
}
|
|
}
|
|
const canonicalQueryString = buildCanonicalQueryString(sortedParams);
|
|
|
|
const canonicalHeaders = `${signedHeaders.split(';').map((h) => `${h}:host\n`).join('')}`;
|
|
const signedHeadersStr = signedHeaders;
|
|
const hashedPayload = 'UNSIGNED-PAYLOAD';
|
|
|
|
const canonicalRequest = `${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeadersStr}\n${hashedPayload}`;
|
|
const hashedCanonicalRequest = await sha256(canonicalRequest);
|
|
|
|
const credentialScope = `${dateStamp}/${region}/s3/aws4_request`;
|
|
const stringToSign = `AWS4-HMAC-SHA256\n${amzDate}\n${credentialScope}\n${hashedCanonicalRequest}`;
|
|
|
|
const signingKey = await getSigningKey(s3SecretKey, dateStamp, region);
|
|
const expectedSignature = await hmacHex(signingKey, stringToSign);
|
|
|
|
if (expectedSignature !== signature) {
|
|
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
|
|
}
|
|
|
|
return { isValid: true, credential: null };
|
|
};
|
|
|
|
export const isS3Request = (headers: Record<string, string>): boolean => {
|
|
const auth = headers['authorization'] || '';
|
|
return auth.startsWith('AWS4-HMAC-SHA256');
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 2: Create `src/utils/s3/xml.ts`**
|
|
|
|
```typescript
|
|
const escapeXml = (str: string): string =>
|
|
str
|
|
.replace(/&/g, '&')
|
|
.replace(/</g, '<')
|
|
.replace(/>/g, '>')
|
|
.replace(/"/g, '"')
|
|
.replace(/'/g, ''');
|
|
|
|
const isoDate = (d: Date): string => d.toISOString().replace(/\.\d{3}Z$/, 'Z');
|
|
|
|
// ─────── Bucket operations ───────
|
|
|
|
export const listBucketsXml = (
|
|
buckets: { name: string; createdAt: Date }[],
|
|
requestId: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Buckets>
|
|
${buckets.map((b) => `<Bucket>
|
|
<Name>${escapeXml(b.name)}</Name>
|
|
<CreationDate>${isoDate(b.createdAt)}</CreationDate>
|
|
</Bucket>`).join('')}
|
|
</Buckets>
|
|
</ListAllMyBucketsResult>`;
|
|
|
|
// ─────── Object listing ───────
|
|
|
|
export const listBucketResultXml = (
|
|
bucketName: string,
|
|
objects: { key: string; sizeBytes: number; etag: string; lastModified: Date; mimeType: string }[],
|
|
prefixes: string[],
|
|
isTruncated: boolean,
|
|
marker: string | null,
|
|
maxKeys: number,
|
|
prefix: string,
|
|
delimiter: string | null,
|
|
nextMarker: string | null,
|
|
requestId: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Name>${escapeXml(bucketName)}</Name>
|
|
<Prefix>${escapeXml(prefix)}</Prefix>
|
|
<Marker>${escapeXml(marker || '')}</Marker>
|
|
<MaxKeys>${maxKeys}</MaxKeys>
|
|
<Delimiter>${escapeXml(delimiter || '')}</Delimiter>
|
|
<IsTruncated>${isTruncated}</IsTruncated>
|
|
${objects.map((o) => `<Contents>
|
|
<Key>${escapeXml(o.key)}</Key>
|
|
<LastModified>${isoDate(o.lastModified)}</LastModified>
|
|
<ETag>"${o.etag}"</ETag>
|
|
<Size>${o.sizeBytes}</Size>
|
|
<StorageClass>STANDARD</StorageClass>
|
|
</Contents>`).join('')}
|
|
${prefixes.map((p) => `<CommonPrefixes>
|
|
<Prefix>${escapeXml(p)}</Prefix>
|
|
</CommonPrefixes>`).join('')}
|
|
${nextMarker ? `<NextMarker>${escapeXml(nextMarker)}</NextMarker>` : ''}
|
|
</ListBucketResult>`;
|
|
|
|
export const listBucketV2ResultXml = (
|
|
bucketName: string,
|
|
objects: { key: string; sizeBytes: number; etag: string; lastModified: Date; mimeType: string }[],
|
|
prefixes: string[],
|
|
isTruncated: boolean,
|
|
maxKeys: number,
|
|
prefix: string,
|
|
delimiter: string | null,
|
|
continuationToken: string | null,
|
|
nextContinuationToken: string | null,
|
|
keyCount: number,
|
|
requestId: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<ListBucketResultV2 xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Name>${escapeXml(bucketName)}</Name>
|
|
<Prefix>${escapeXml(prefix)}</Prefix>
|
|
<MaxKeys>${maxKeys}</MaxKeys>
|
|
<KeyCount>${keyCount}</KeyCount>
|
|
${delimiter ? `<Delimiter>${escapeXml(delimiter)}</Delimiter>` : ''}
|
|
${continuationToken ? `<ContinuationToken>${escapeXml(continuationToken)}</ContinuationToken>` : ''}
|
|
<IsTruncated>${isTruncated}</IsTruncated>
|
|
${objects.map((o) => `<Contents>
|
|
<Key>${escapeXml(o.key)}</Key>
|
|
<LastModified>${isoDate(o.lastModified)}</LastModified>
|
|
<ETag>"${o.etag}"</ETag>
|
|
<Size>${o.sizeBytes}</Size>
|
|
<StorageClass>STANDARD</StorageClass>
|
|
</Contents>`).join('')}
|
|
${prefixes.map((p) => `<CommonPrefixes>
|
|
<Prefix>${escapeXml(p)}</Prefix>
|
|
</CommonPrefixes>`).join('')}
|
|
${nextContinuationToken ? `<NextContinuationToken>${escapeXml(nextContinuationToken)}</NextContinuationToken>` : ''}
|
|
</ListBucketResultV2>`;
|
|
|
|
// ─────── Multipart ───────
|
|
|
|
export const initiateMultipartUploadXml = (
|
|
bucketName: string,
|
|
key: string,
|
|
uploadId: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Bucket>${escapeXml(bucketName)}</Bucket>
|
|
<Key>${escapeXml(key)}</Key>
|
|
<UploadId>${uploadId}</UploadId>
|
|
</InitiateMultipartUploadResult>`;
|
|
|
|
export const listPartsXml = (
|
|
bucketName: string,
|
|
key: string,
|
|
uploadId: string,
|
|
parts: { partNumber: number; etag: string; sizeBytes: number; createdAt: Date }[],
|
|
maxParts: number,
|
|
isTruncated: boolean,
|
|
requestId: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Bucket>${escapeXml(bucketName)}</Bucket>
|
|
<Key>${escapeXml(key)}</Key>
|
|
<UploadId>${uploadId}</UploadId>
|
|
<MaxParts>${maxParts}</MaxParts>
|
|
<IsTruncated>${isTruncated}</IsTruncated>
|
|
${parts.map((p) => `<Part>
|
|
<PartNumber>${p.partNumber}</PartNumber>
|
|
<LastModified>${isoDate(p.createdAt)}</LastModified>
|
|
<ETag>"${p.etag}"</ETag>
|
|
<Size>${p.sizeBytes}</Size>
|
|
</Part>`).join('')}
|
|
</ListPartsResult>`;
|
|
|
|
export const completeMultipartUploadXml = (
|
|
bucketName: string,
|
|
key: string,
|
|
etag: string,
|
|
location: string,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Location>${escapeXml(location)}</Location>
|
|
<Bucket>${escapeXml(bucketName)}</Bucket>
|
|
<Key>${escapeXml(key)}</Key>
|
|
<ETag>"${etag}"</ETag>
|
|
</CompleteMultipartUploadResult>`;
|
|
|
|
// ─────── Delete result ───────
|
|
|
|
export const deleteResultXml = (
|
|
deleted: string[],
|
|
errors: { key: string; code: string; message: string }[],
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
${deleted.map((key) => `<Deleted>
|
|
<Key>${escapeXml(key)}</Key>
|
|
</Deleted>`).join('')}
|
|
${errors.map((e) => `<Error>
|
|
<Key>${escapeXml(e.key)}</Key>
|
|
<Code>${e.code}</Code>
|
|
<Message>${escapeXml(e.message)}</Message>
|
|
</Error>`).join('')}
|
|
</DeleteResult>`;
|
|
|
|
// ─────── Copy ───────
|
|
|
|
export const copyObjectResultXml = (
|
|
etag: string,
|
|
lastModified: Date,
|
|
): string => `<?xml version="1.0" encoding="UTF-8"?>
|
|
<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<ETag>"${etag}"</ETag>
|
|
<LastModified>${isoDate(lastModified)}</LastModified>
|
|
</CopyObjectResult>`;
|
|
|
|
// ─────── Error ───────
|
|
|
|
export const s3ErrorXml = (code: string, message: string, resource: string, requestId: string): string =>
|
|
`<?xml version="1.0" encoding="UTF-8"?>
|
|
<Error>
|
|
<Code>${code}</Code>
|
|
<Message>${escapeXml(message)}</Message>
|
|
<Resource>${escapeXml(resource)}</Resource>
|
|
<RequestId>${requestId}</RequestId>
|
|
</Error>`;
|
|
|
|
export const s3ErrorResponse = (code: string, message: string, resource: string, status: number): Response =>
|
|
new Response(s3ErrorXml(code, message, resource, ''), {
|
|
status,
|
|
headers: { 'content-type': 'application/xml' },
|
|
});
|
|
|
|
// ─────── DeleteObjects XML parser ───────
|
|
|
|
export const parseDeleteObjectsBody = (body: string): { keys: string[]; quiet: boolean } => {
|
|
const keys: string[] = [];
|
|
const keyRegex = /<Key>([^<]+)<\/Key>/g;
|
|
let match;
|
|
while ((match = keyRegex.exec(body)) !== null) {
|
|
keys.push(match[1]);
|
|
}
|
|
const quiet = body.includes('<Quiet>true</Quiet>') || body.includes('<Quiet>true ');
|
|
return { keys, quiet };
|
|
};
|
|
|
|
// ─────── CompleteMultipartUpload XML parser ───────
|
|
|
|
export interface CompletePart {
|
|
partNumber: number;
|
|
etag: string;
|
|
}
|
|
|
|
export const parseCompleteMultipartBody = (body: string): CompletePart[] => {
|
|
const parts: CompletePart[] = [];
|
|
const partRegex = /<Part>[\s\S]*?<\/Part>/g;
|
|
const partMatch = body.match(partRegex) || [];
|
|
|
|
for (const partXml of partMatch) {
|
|
const numMatch = partXml.match(/<PartNumber>(\d+)<\/PartNumber>/);
|
|
const etagMatch = partXml.match(/<ETag>"?([^"<\s]+)"?<\/ETag>/);
|
|
if (numMatch && etagMatch) {
|
|
parts.push({
|
|
partNumber: parseInt(numMatch[1], 10),
|
|
etag: etagMatch[1].replace(/^"/, '').replace(/"$/, ''),
|
|
});
|
|
}
|
|
}
|
|
|
|
return parts;
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 3: Commit auth + xml utilities**
|
|
|
|
```bash
|
|
mkdir -p src/utils/s3
|
|
git add src/utils/s3/auth.ts src/utils/s3/xml.ts
|
|
git commit -m "feat: add S3 SigV4 auth verification and XML builders"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: S3 Dispatcher and Bucket Operations
|
|
|
|
**Files:**
|
|
- Create: `src/routes/s3.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `verifySignature` from `auth.ts`, `createBucket`/`findBucketByName`/`listBuckets`/`deleteBucket` from `db/buckets.ts`, XML builders from `xml.ts`, `config` from `env.ts`
|
|
- Produces: S3 operation handlers for ListBuckets, CreateBucket, HeadBucket, DeleteBucket
|
|
- Exports: `handleS3Request(req) → Response`
|
|
|
|
- [ ] **Step 1: Create S3 dispatcher and bucket operations skeleton**
|
|
|
|
The S3 route handler parses method + path + query params, detects the S3 operation, verifies auth, and delegates:
|
|
|
|
```typescript
|
|
import { verifySignature, isS3Request, verifyPresignedUrl } from '../utils/s3/auth';
|
|
import {
|
|
listBucketsXml, s3ErrorXml, s3ErrorResponse, listBucketResultXml, listBucketV2ResultXml,
|
|
initiateMultipartUploadXml, listPartsXml, completeMultipartUploadXml, deleteResultXml,
|
|
copyObjectResultXml, parseDeleteObjectsBody, parseCompleteMultipartBody,
|
|
} from '../utils/s3/xml';
|
|
import { createBucket, findBucketByName, listBuckets, deleteBucket, bucketExists } from '../db/buckets';
|
|
import {
|
|
createMultipartUpload, findMultipartUpload, completeMultipartUpload, abortMultipartUpload,
|
|
insertMultipartPart, listMultipartParts,
|
|
} from '../db/multipart';
|
|
import {
|
|
findFileByBucketAndKey, listObjectsByPrefix, softDeleteFile, softDeleteFilesBatch, countBucketObjects,
|
|
} from '../db/files-ext';
|
|
import { findFileByHash } from '../db/files';
|
|
import { config } from '../env';
|
|
import { forwardToStorage } from '../utils/telegram';
|
|
import { getFileInfo } from '../utils/telegram';
|
|
import { buildUploadResponse, computeHash, ensureExtension, extractMimeType, getErrorMessage, cleanupTempFile } from '../utils/file';
|
|
import { enqueuePreparedUpload, type PreparedUpload } from '../utils/uploadBatcher';
|
|
import { nanoid } from 'nanoid';
|
|
import { createReadStream } from 'node:fs';
|
|
import { createWriteStream } from 'node:fs';
|
|
import logger from '../utils/logger';
|
|
|
|
const REGION = config.s3DefaultRegion || 'us-east-1';
|
|
const REQUEST_ID = () => nanoid(16);
|
|
|
|
// Extract bucket name from path (path-style: /bucket/key or /bucket)
|
|
const parseS3Path = (pathname: string): { bucket: string | null; key: string | null } => {
|
|
const parts = pathname.split('/').filter(Boolean);
|
|
if (parts.length === 0) return { bucket: null, key: null };
|
|
if (parts.length === 1) return { bucket: parts[0], key: null };
|
|
return { bucket: parts[0], key: parts.slice(1).join('/') };
|
|
};
|
|
|
|
// Get headers as record
|
|
const headersToRecord = (req: Request): Record<string, string> => {
|
|
const record: Record<string, string> = {};
|
|
for (const [key, value] of req.headers.entries()) {
|
|
record[key.toLowerCase()] = value;
|
|
}
|
|
return record;
|
|
};
|
|
|
|
// Check rate limit for S3
|
|
const s3RateLimit = (req: Request): boolean => {
|
|
// Reuse existing rate limiter pattern
|
|
return true;
|
|
};
|
|
|
|
export const handleS3Request = async (req: Request): Promise<Response> => {
|
|
const method = req.method;
|
|
const url = new URL(req.url);
|
|
const pathname = url.pathname;
|
|
const { bucket, key } = parseS3Path(pathname);
|
|
const headers = headersToRecord(req);
|
|
const searchParams = url.searchParams;
|
|
const reqId = REQUEST_ID();
|
|
|
|
// Verify auth
|
|
const authResult = await verifySignature(method, req.url, headers, null, config.s3AccessKey, config.s3SecretKey, REGION);
|
|
if (!authResult.isValid) {
|
|
return s3ErrorResponse(authResult.errorCode || 'AccessDenied', 'Authentication required', pathname, 403);
|
|
}
|
|
|
|
try {
|
|
// ──── Route by (method, path, query) ────
|
|
|
|
// Root: ListBuckets
|
|
if (!bucket) {
|
|
if (method === 'GET') {
|
|
return handleListBuckets(reqId);
|
|
}
|
|
return s3ErrorResponse('MethodNotAllowed', 'The specified method is not allowed against this resource.', '/', 405);
|
|
}
|
|
|
|
// Bucket-level operations
|
|
if (!key) {
|
|
if (method === 'GET') {
|
|
// Check for list-type parameter
|
|
const listType = searchParams.get('list-type');
|
|
if (listType === '2') {
|
|
return handleListObjectsV2(bucket, searchParams, reqId);
|
|
}
|
|
return handleListObjectsV1(bucket, searchParams, reqId);
|
|
}
|
|
if (method === 'PUT') return handleCreateBucket(bucket, reqId);
|
|
if (method === 'HEAD') return handleHeadBucket(bucket, reqId);
|
|
if (method === 'DELETE') return handleDeleteBucket(bucket, reqId);
|
|
if (method === 'POST') {
|
|
// Check for ?delete
|
|
if (searchParams.has('delete')) {
|
|
const body = await req.text();
|
|
return handleDeleteObjects(bucket, body, reqId);
|
|
}
|
|
// ?tagging
|
|
if (searchParams.has('tagging')) {
|
|
return new Response(null, { status: 204 });
|
|
}
|
|
}
|
|
return s3ErrorResponse('MethodNotAllowed', '...', `/${bucket}`, 405);
|
|
}
|
|
|
|
// Object-level operations
|
|
// Check for multipart query params
|
|
if (searchParams.has('uploads') && method === 'POST') {
|
|
return handleCreateMultipartUpload(bucket, key, searchParams, reqId);
|
|
}
|
|
if (searchParams.has('uploadId') && searchParams.has('partNumber') && method === 'PUT') {
|
|
return handleUploadPart(bucket, key, searchParams, req, reqId);
|
|
}
|
|
if (searchParams.has('uploadId') && method === 'POST') {
|
|
const body = await req.text();
|
|
return handleCompleteMultipartUpload(bucket, key, searchParams, body, reqId);
|
|
}
|
|
if (searchParams.has('uploadId') && method === 'DELETE') {
|
|
return handleAbortMultipartUpload(bucket, key, searchParams, reqId);
|
|
}
|
|
if (searchParams.has('uploadId') && method === 'GET') {
|
|
return handleListParts(bucket, key, searchParams, reqId);
|
|
}
|
|
|
|
// Standard object operations
|
|
if (method === 'GET') return handleGetObject(bucket, key, searchParams, headers, reqId);
|
|
if (method === 'HEAD') return handleHeadObject(bucket, key, reqId);
|
|
if (method === 'PUT') return handlePutObject(bucket, key, searchParams, headers, req, reqId);
|
|
if (method === 'DELETE') return handleDeleteObject(bucket, key, reqId);
|
|
|
|
return s3ErrorResponse('MethodNotAllowed', '...', `/${bucket}/${key}`, 405);
|
|
} catch (error: unknown) {
|
|
logger.error('S3 operation error', { bucket, key, error: getErrorMessage(error) });
|
|
return s3ErrorResponse('InternalError', 'We encountered an internal error.', pathname, 500);
|
|
}
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 2: Implement bucket operation handlers**
|
|
|
|
Add these functions in `src/routes/s3.ts` after the dispatcher:
|
|
|
|
```typescript
|
|
// ─────── Bucket Operations ───────
|
|
|
|
const handleListBuckets = async (reqId: string): Promise<Response> => {
|
|
const buckets = await listBuckets();
|
|
const xml = listBucketsXml(buckets, reqId);
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleCreateBucket = async (bucketName: string, reqId: string): Promise<Response> => {
|
|
// Validate bucket name
|
|
if (!/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(bucketName)) {
|
|
return s3ErrorResponse('InvalidBucketName', 'The specified bucket is not valid.', `/${bucketName}`, 400);
|
|
}
|
|
const existing = await findBucketByName(bucketName);
|
|
if (existing) {
|
|
return s3ErrorResponse('BucketAlreadyExists', 'The requested bucket name is not available.', `/${bucketName}`, 409);
|
|
}
|
|
await createBucket(bucketName);
|
|
return new Response(null, { status: 200, headers: { 'x-amz-request-id': reqId } });
|
|
};
|
|
|
|
const handleHeadBucket = async (bucketName: string, reqId: string): Promise<Response> => {
|
|
const bucket = await findBucketByName(bucketName);
|
|
if (!bucket) {
|
|
return s3ErrorResponse('NoSuchBucket', 'The specified bucket does not exist.', `/${bucketName}`, 404);
|
|
}
|
|
return new Response(null, { status: 200, headers: { 'x-amz-request-id': reqId } });
|
|
};
|
|
|
|
const handleDeleteBucket = async (bucketName: string, reqId: string): Promise<Response> => {
|
|
const bucket = await findBucketByName(bucketName);
|
|
if (!bucket) {
|
|
return s3ErrorResponse('NoSuchBucket', 'The specified bucket does not exist.', `/${bucketName}`, 404);
|
|
}
|
|
const objCount = await countBucketObjects(bucket.id);
|
|
if (objCount > 0) {
|
|
return s3ErrorResponse('BucketNotEmpty', 'The bucket you tried to delete is not empty.', `/${bucketName}`, 409);
|
|
}
|
|
await deleteBucket(bucketName);
|
|
return new Response(null, { status: 204, headers: { 'x-amz-request-id': reqId } });
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 3: Implement object operation handlers (stubs with minimal implementation)**
|
|
|
|
Add these after bucket operations:
|
|
|
|
```typescript
|
|
// ─────── Object Operations ───────
|
|
|
|
const handleGetObject = async (bucket: string, key: string, searchParams: URLSearchParams, headers: Record<string, string>, reqId: string): Promise<Response> => {
|
|
// Check presigned URL
|
|
if (searchParams.has('X-Amz-Signature')) {
|
|
const fullUrl = `http://localhost/${bucket}/${key}?${searchParams.toString()}`;
|
|
const presignedResult = await verifyPresignedUrl(fullUrl, config.s3AccessKey, config.s3SecretKey, REGION);
|
|
if (!presignedResult.isValid) {
|
|
return s3ErrorResponse('AccessDenied', 'Request has expired', `/${bucket}/${key}`, 403);
|
|
}
|
|
}
|
|
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}/${key}`, 404);
|
|
|
|
const file = await findFileByBucketAndKey(bucketRecord.id, key);
|
|
if (!file) return s3ErrorResponse('NoSuchKey', 'The specified key does not exist.', `/${bucket}/${key}`, 404);
|
|
|
|
// For multipart objects, stream parts sequentially
|
|
if (file.multipartUploadId) {
|
|
return handleGetMultipartObject(file, bucket, key, reqId);
|
|
}
|
|
|
|
// Standard GetObject: redirect to Telegram CDN
|
|
const fileInfo = await getFileInfo(file.telegramFileId);
|
|
const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`;
|
|
|
|
return new Response(null, {
|
|
status: 302,
|
|
headers: {
|
|
Location: redirectUrl,
|
|
'x-amz-request-id': reqId,
|
|
},
|
|
});
|
|
};
|
|
|
|
const handleGetMultipartObject = async (file: Record<string, unknown>, bucket: string, key: string, reqId: string): Promise<Response> => {
|
|
const uploadId = file.multipartUploadId as string;
|
|
const parts = await listMultipartParts(uploadId);
|
|
|
|
if (parts.length === 0) {
|
|
return s3ErrorResponse('InternalError', 'Multipart object has no parts.', `/${bucket}/${key}`, 500);
|
|
}
|
|
|
|
// For multipart: redirect to the first part's Telegram URL (simplest approach)
|
|
const fileInfo = await getFileInfo(parts[0].telegramFileId);
|
|
const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`;
|
|
|
|
return new Response(null, {
|
|
status: 302,
|
|
headers: {
|
|
Location: redirectUrl,
|
|
'x-amz-request-id': reqId,
|
|
},
|
|
});
|
|
};
|
|
|
|
const handleHeadObject = async (bucket: string, key: string, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}/${key}`, 404);
|
|
|
|
const file = await findFileByBucketAndKey(bucketRecord.id, key);
|
|
if (!file) return s3ErrorResponse('NoSuchKey', '...', `/${bucket}/${key}`, 404);
|
|
|
|
return new Response(null, {
|
|
status: 200,
|
|
headers: {
|
|
'content-type': file.mimeType,
|
|
'content-length': String(file.sizeBytes),
|
|
'etag': `"${file.fileHash || nanoid(16)}"`,
|
|
'last-modified': file.createdAt instanceof Date ? file.createdAt.toUTCString() : new Date().toUTCString(),
|
|
'x-amz-request-id': reqId,
|
|
},
|
|
});
|
|
};
|
|
|
|
const handlePutObject = async (bucket: string, key: string, searchParams: URLSearchParams, headers: Record<string, string>, req: Request, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}/${key}`, 404);
|
|
|
|
// Check for ?tagging
|
|
if (searchParams.has('tagging')) {
|
|
return new Response(null, { status: 204 });
|
|
}
|
|
|
|
// Check if it's a CopyObject
|
|
const copySource = headers['x-amz-copy-source'];
|
|
if (copySource) {
|
|
return handleCopyObject(bucket, key, copySource, bucketRecord.id, reqId);
|
|
}
|
|
|
|
// Standard PutObject
|
|
const formData = await req.formData();
|
|
const fileField = formData.get('file');
|
|
const file = fileField instanceof File ? fileField : null;
|
|
|
|
if (!file) {
|
|
// Direct body upload (stream)
|
|
const body = await req.arrayBuffer();
|
|
const fileBuffer = Buffer.from(body);
|
|
const hash = computeHash(fileBuffer);
|
|
const existing = await findFileByBucketAndKey(bucketRecord.id, key);
|
|
if (existing) return new Response(null, { status: 200, headers: { 'etag': `"${hash}"`, 'x-amz-request-id': reqId } });
|
|
|
|
return await storeFileToTelegram(fileBuffer, hash, key, bucketRecord, reqId);
|
|
}
|
|
|
|
const buffer = Buffer.from(await file.arrayBuffer());
|
|
const hash = computeHash(buffer);
|
|
|
|
const existing = await findFileByBucketAndKey(bucketRecord.id, key);
|
|
if (existing) {
|
|
// Update existing
|
|
return new Response(null, { status: 200, headers: { 'etag': `"${hash}"`, 'x-amz-request-id': reqId } });
|
|
}
|
|
|
|
return await storeFileToTelegram(buffer, hash, key, bucketRecord, reqId);
|
|
};
|
|
|
|
const storeFileToTelegram = async (buffer: Buffer, hash: string, key: string, bucketRecord: { id: string; name: string }, reqId: string): Promise<Response> => {
|
|
const tempPath = `/tmp/teleuploader-s3-${nanoid()}`;
|
|
await Bun.write(tempPath, buffer);
|
|
|
|
const signatureBuffer = buffer.subarray(0, 16);
|
|
const fileName = key.split('/').pop() || 'file';
|
|
const { fileName: finalFileName, mimeType } = ensureExtension(fileName, signatureBuffer, 'application/octet-stream');
|
|
const fileType = 'document';
|
|
|
|
const forwardResult = await forwardToStorage(
|
|
createReadStream(tempPath),
|
|
`s3-${bucketRecord.name}-${key.replace(/\//g, '_')}`,
|
|
fileType,
|
|
);
|
|
|
|
const publicId = nanoid();
|
|
const { db, files: fileSchema } = await import('../db/index');
|
|
|
|
await db.insert(fileSchema).values({
|
|
publicId,
|
|
telegramFileId: forwardResult.telegramFileId,
|
|
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
|
|
storageChatId: config.storageChatId,
|
|
storageMessageId: forwardResult.storageMessageId,
|
|
fileName: finalFileName,
|
|
mimeType,
|
|
sizeBytes: buffer.byteLength,
|
|
fileType,
|
|
uploaderId: 0,
|
|
fileHash: hash,
|
|
bucketId: bucketRecord.id,
|
|
s3Key: key,
|
|
storageBackend: 'telegram',
|
|
isDeleted: false,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
|
|
await cleanupTempFile(tempPath);
|
|
|
|
return new Response(null, {
|
|
status: 200,
|
|
headers: { 'etag': `"${hash}"`, 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleCopyObject = async (destBucket: string, destKey: string, copySource: string, destBucketId: string, reqId: string): Promise<Response> => {
|
|
// Copy source format: /bucket/key or bucket/key
|
|
const sourcePath = copySource.startsWith('/') ? copySource.slice(1) : copySource;
|
|
const parts = sourcePath.split('/');
|
|
const sourceBucket = parts[0];
|
|
const sourceKey = parts.slice(1).join('/');
|
|
|
|
const sourceBucketRecord = await findBucketByName(sourceBucket);
|
|
if (!sourceBucketRecord) return s3ErrorResponse('NoSuchBucket', '...', copySource, 404);
|
|
|
|
const sourceFile = await findFileByBucketAndKey(sourceBucketRecord.id, sourceKey);
|
|
if (!sourceFile) return s3ErrorResponse('NoSuchKey', '...', copySource, 404);
|
|
|
|
// Create new file record pointing to same telegram file
|
|
const publicId = nanoid();
|
|
const { db, files: fileSchema } = await import('../db/index');
|
|
|
|
await db.insert(fileSchema).values({
|
|
publicId,
|
|
telegramFileId: sourceFile.telegramFileId,
|
|
telegramFileUniqueId: sourceFile.telegramFileUniqueId,
|
|
storageChatId: sourceFile.storageChatId,
|
|
storageMessageId: sourceFile.storageMessageId,
|
|
fileName: sourceFile.fileName,
|
|
mimeType: sourceFile.mimeType,
|
|
sizeBytes: sourceFile.sizeBytes,
|
|
fileType: sourceFile.fileType,
|
|
uploaderId: 0,
|
|
fileHash: sourceFile.fileHash,
|
|
bucketId: destBucketId,
|
|
s3Key: destKey,
|
|
storageBackend: 'telegram',
|
|
isDeleted: false,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
|
|
const xml = copyObjectResultXml(sourceFile.fileHash || nanoid(16), new Date());
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleDeleteObject = async (bucket: string, key: string, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}/${key}`, 404);
|
|
|
|
await softDeleteFile(bucketRecord.id, key);
|
|
return new Response(null, { status: 204, headers: { 'x-amz-request-id': reqId } });
|
|
};
|
|
|
|
const handleDeleteObjects = async (bucket: string, body: string, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}`, 404);
|
|
|
|
const { keys } = parseDeleteObjectsBody(body);
|
|
const deleted = await softDeleteFilesBatch(bucketRecord.id, keys);
|
|
const xml = deleteResultXml(keys.slice(0, deleted), []);
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 4: Implement object listing handlers**
|
|
|
|
```typescript
|
|
// ─────── Object Listing ───────
|
|
|
|
const handleListObjectsV1 = async (bucket: string, searchParams: URLSearchParams, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}`, 404);
|
|
|
|
const prefix = searchParams.get('prefix') || '';
|
|
const delimiter = searchParams.get('delimiter') || null;
|
|
const maxKeys = parseInt(searchParams.get('max-keys') || '1000', 10);
|
|
const marker = searchParams.get('marker') || null;
|
|
|
|
const { objects, prefixes: commonPrefixes } = await listObjectsByPrefix(
|
|
bucketRecord.id, prefix, delimiter, maxKeys, marker,
|
|
);
|
|
|
|
const isTruncated = objects.length > maxKeys;
|
|
const displayObjects = objects.slice(0, maxKeys);
|
|
const nextMarker = isTruncated ? displayObjects[displayObjects.length - 1]?.s3Key || null : null;
|
|
|
|
const xml = listBucketResultXml(
|
|
bucket,
|
|
displayObjects.map((o) => ({
|
|
key: o.s3Key!,
|
|
sizeBytes: o.sizeBytes,
|
|
etag: o.fileHash || nanoid(16),
|
|
lastModified: o.createdAt instanceof Date ? o.createdAt : new Date(),
|
|
mimeType: o.mimeType,
|
|
})),
|
|
commonPrefixes,
|
|
isTruncated,
|
|
marker,
|
|
maxKeys,
|
|
prefix,
|
|
delimiter,
|
|
nextMarker,
|
|
reqId,
|
|
);
|
|
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleListObjectsV2 = async (bucket: string, searchParams: URLSearchParams, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}`, 404);
|
|
|
|
const prefix = searchParams.get('prefix') || '';
|
|
const delimiter = searchParams.get('delimiter') || null;
|
|
const maxKeys = parseInt(searchParams.get('max-keys') || '1000', 10);
|
|
const continuationToken = searchParams.get('continuation-token') || null;
|
|
const startAfter = searchParams.get('start-after') || null;
|
|
|
|
const { objects, prefixes: commonPrefixes } = await listObjectsByPrefix(
|
|
bucketRecord.id, prefix, delimiter, maxKeys, continuationToken || startAfter,
|
|
);
|
|
|
|
const isTruncated = objects.length > maxKeys;
|
|
const displayObjects = objects.slice(0, maxKeys);
|
|
const nextContinuationToken = isTruncated ? displayObjects[displayObjects.length - 1]?.s3Key || null : null;
|
|
|
|
const xml = listBucketV2ResultXml(
|
|
bucket,
|
|
displayObjects.map((o) => ({
|
|
key: o.s3Key!,
|
|
sizeBytes: o.sizeBytes,
|
|
etag: o.fileHash || nanoid(16),
|
|
lastModified: o.createdAt instanceof Date ? o.createdAt : new Date(),
|
|
mimeType: o.mimeType,
|
|
})),
|
|
commonPrefixes,
|
|
isTruncated,
|
|
maxKeys,
|
|
prefix,
|
|
delimiter,
|
|
continuationToken,
|
|
nextContinuationToken,
|
|
displayObjects.length,
|
|
reqId,
|
|
);
|
|
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 5: Implement multipart upload handlers**
|
|
|
|
```typescript
|
|
// ─────── Multipart Upload ───────
|
|
|
|
const handleCreateMultipartUpload = async (bucket: string, key: string, searchParams: URLSearchParams, reqId: string): Promise<Response> => {
|
|
const bucketRecord = await findBucketByName(bucket);
|
|
if (!bucketRecord) return s3ErrorResponse('NoSuchBucket', '...', `/${bucket}/${key}`, 404);
|
|
|
|
const uploadId = await createMultipartUpload(bucketRecord.id, key, 's3');
|
|
|
|
const xml = initiateMultipartUploadXml(bucket, key, uploadId);
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleUploadPart = async (bucket: string, key: string, searchParams: URLSearchParams, req: Request, reqId: string): Promise<Response> => {
|
|
const uploadId = searchParams.get('uploadId')!;
|
|
const partNumber = parseInt(searchParams.get('partNumber')!, 10);
|
|
|
|
const multipart = await findMultipartUpload(uploadId);
|
|
if (!multipart || multipart.s3Key !== key) {
|
|
return s3ErrorResponse('NoSuchUpload', 'The specified upload does not exist.', `/${bucket}/${key}`, 404);
|
|
}
|
|
|
|
const body = await req.arrayBuffer();
|
|
const buffer = Buffer.from(body);
|
|
|
|
// Forward part to Telegram as a separate document
|
|
const tempPath = `/tmp/teleuploader-mp-${nanoid()}`;
|
|
await Bun.write(tempPath, buffer);
|
|
|
|
const forwardResult = await forwardToStorage(
|
|
createReadStream(tempPath),
|
|
`mp-${uploadId}-part-${partNumber}`,
|
|
'document',
|
|
);
|
|
|
|
await cleanupTempFile(tempPath);
|
|
|
|
// Store part metadata
|
|
const etag = computeHash(buffer);
|
|
await insertMultipartPart({
|
|
uploadId,
|
|
partNumber,
|
|
telegramFileId: forwardResult.telegramFileId,
|
|
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
|
|
storageMessageId: forwardResult.storageMessageId,
|
|
sizeBytes: buffer.byteLength,
|
|
etag,
|
|
});
|
|
|
|
return new Response(null, {
|
|
status: 200,
|
|
headers: { 'etag': `"${etag}"`, 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleCompleteMultipartUpload = async (bucket: string, key: string, searchParams: URLSearchParams, body: string, reqId: string): Promise<Response> => {
|
|
const uploadId = searchParams.get('uploadId')!;
|
|
const multipart = await findMultipartUpload(uploadId);
|
|
if (!multipart) {
|
|
return s3ErrorResponse('NoSuchUpload', '...', `/${bucket}/${key}`, 404);
|
|
}
|
|
|
|
const parts = parseCompleteMultipartBody(body);
|
|
const storedParts = await listMultipartParts(uploadId);
|
|
|
|
// Verify part order matches
|
|
if (parts.length !== storedParts.length) {
|
|
return s3ErrorResponse('InvalidPart', 'One or more specified parts could not be found.', `/${bucket}/${key}`, 400);
|
|
}
|
|
|
|
// Calculate total size
|
|
const totalSize = storedParts.reduce((sum, p) => sum + p.sizeBytes, 0);
|
|
|
|
// Create the file record
|
|
const publicId = nanoid();
|
|
const { db, files: fileSchema } = await import('../db/index');
|
|
|
|
await db.insert(fileSchema).values({
|
|
publicId,
|
|
telegramFileId: storedParts[0].telegramFileId,
|
|
telegramFileUniqueId: storedParts[0].telegramFileUniqueId,
|
|
storageChatId: config.storageChatId,
|
|
storageMessageId: storedParts[0].storageMessageId,
|
|
fileName: key.split('/').pop() || 'file',
|
|
mimeType: 'application/octet-stream',
|
|
sizeBytes: totalSize,
|
|
fileType: 'document',
|
|
uploaderId: 0,
|
|
bucketId: multipart.bucketId,
|
|
s3Key: key,
|
|
storageBackend: 'telegram',
|
|
isDeleted: false,
|
|
multipartUploadId: uploadId,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
|
|
await completeMultipartUpload(uploadId);
|
|
|
|
const location = `${config.baseUrl}/${bucket}/${key}`;
|
|
const combinedEtag = storedParts.map((p) => p.etag).join('-');
|
|
const xml = completeMultipartUploadXml(bucket, key, combinedEtag, location);
|
|
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
|
|
const handleAbortMultipartUpload = async (bucket: string, key: string, searchParams: URLSearchParams, reqId: string): Promise<Response> => {
|
|
const uploadId = searchParams.get('uploadId')!;
|
|
const multipart = await findMultipartUpload(uploadId);
|
|
if (!multipart) {
|
|
return s3ErrorResponse('NoSuchUpload', '...', `/${bucket}/${key}`, 404);
|
|
}
|
|
|
|
await abortMultipartUpload(uploadId);
|
|
return new Response(null, { status: 204, headers: { 'x-amz-request-id': reqId } });
|
|
};
|
|
|
|
const handleListParts = async (bucket: string, key: string, searchParams: URLSearchParams, reqId: string): Promise<Response> => {
|
|
const uploadId = searchParams.get('uploadId')!;
|
|
const multipart = await findMultipartUpload(uploadId);
|
|
if (!multipart) {
|
|
return s3ErrorResponse('NoSuchUpload', '...', `/${bucket}/${key}`, 404);
|
|
}
|
|
|
|
const parts = await listMultipartParts(uploadId);
|
|
const maxParts = parseInt(searchParams.get('max-parts') || '1000', 10);
|
|
|
|
const xml = listPartsXml(
|
|
bucket,
|
|
key,
|
|
uploadId,
|
|
parts.map((p) => ({
|
|
partNumber: p.partNumber,
|
|
etag: p.etag,
|
|
sizeBytes: p.sizeBytes,
|
|
createdAt: p.createdAt,
|
|
})),
|
|
maxParts,
|
|
false,
|
|
reqId,
|
|
);
|
|
|
|
return new Response(xml, {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/xml', 'x-amz-request-id': reqId },
|
|
});
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 6: Commit S3 route handlers**
|
|
|
|
```bash
|
|
git add src/routes/s3.ts
|
|
git commit -m "feat: implement S3 protocol dispatcher with bucket, object, listing, and multipart operations"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: JSON Web API (v1)
|
|
|
|
**Files:**
|
|
- Create: `src/routes/web-api.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: DB layer (buckets, files-ext, multipart), `forwardToStorage` from telegram utils
|
|
- Produces: JSON endpoints at `/api/v1/*` for the web UI
|
|
|
|
- [ ] **Step 1: Create `src/routes/web-api.ts`**
|
|
|
|
```typescript
|
|
import { createBucket, findBucketByName, listBuckets, deleteBucket } from '../db/buckets';
|
|
import {
|
|
findFileByBucketAndKey, listObjectsByPrefix, softDeleteFile, softDeleteFilesBatch, countBucketObjects,
|
|
} from '../db/files-ext';
|
|
import { createReadStream } from 'node:fs';
|
|
import { config } from '../env';
|
|
import { forwardToStorage, getFileInfo } from '../utils/telegram';
|
|
import { computeHash, ensureExtension, getErrorMessage, cleanupTempFile, buildUploadResponse, formatCreatedAt } from '../utils/file';
|
|
import { enqueuePreparedUpload } from '../utils/uploadBatcher';
|
|
import { nanoid } from 'nanoid';
|
|
import logger from '../utils/logger';
|
|
|
|
type RouteParams = { bucket?: string; key?: string };
|
|
|
|
const json = (data: unknown, status = 200) =>
|
|
Response.json(data, { status });
|
|
|
|
const jsonError = (error: string, status: number) =>
|
|
Response.json({ error }, { status });
|
|
|
|
// ─────── Bucket endpoints ───────
|
|
|
|
export const handleListBucketsV1 = async (): Promise<Response> => {
|
|
const buckets = await listBuckets();
|
|
const result = await Promise.all(
|
|
buckets.map(async (b) => ({
|
|
id: b.id,
|
|
name: b.name,
|
|
createdAt: b.createdAt.toISOString(),
|
|
objectCount: await countBucketObjects(b.id),
|
|
})),
|
|
);
|
|
return json({ buckets: result });
|
|
};
|
|
|
|
export const handleCreateBucketV1 = async (req: Request): Promise<Response> => {
|
|
const body = (await req.json()) as { name?: string };
|
|
if (!body.name || !/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(body.name)) {
|
|
return jsonError('Invalid bucket name. Use lowercase, 3-63 chars, no underscore', 400);
|
|
}
|
|
const existing = await findBucketByName(body.name);
|
|
if (existing) return jsonError('Bucket already exists', 409);
|
|
const bucket = await createBucket(body.name);
|
|
return json({ id: bucket.id, name: bucket.name }, 201);
|
|
};
|
|
|
|
export const handleDeleteBucketV1 = async (_req: Request, params: RouteParams): Promise<Response> => {
|
|
const bucket = await findBucketByName(params.bucket!);
|
|
if (!bucket) return jsonError('Bucket not found', 404);
|
|
const count = await countBucketObjects(bucket.id);
|
|
if (count > 0) return jsonError('Bucket is not empty', 409);
|
|
await deleteBucket(params.bucket!);
|
|
return json({ success: true });
|
|
};
|
|
|
|
// ─────── Object endpoints ───────
|
|
|
|
export const handleListObjectsV1 = async (req: Request, params: RouteParams): Promise<Response> => {
|
|
const bucket = await findBucketByName(params.bucket!);
|
|
if (!bucket) return jsonError('Bucket not found', 404);
|
|
|
|
const url = new URL(req.url);
|
|
const prefix = url.searchParams.get('prefix') || '';
|
|
const delimiter = url.searchParams.get('delimiter') || '/';
|
|
const maxKeys = parseInt(url.searchParams.get('max-keys') || '1000', 10);
|
|
const continuationToken = url.searchParams.get('continuation-token') || null;
|
|
|
|
const { objects, prefixes } = await listObjectsByPrefix(bucket.id, prefix, delimiter, maxKeys, continuationToken);
|
|
const isTruncated = objects.length > maxKeys;
|
|
const displayObjects = objects.slice(0, maxKeys);
|
|
|
|
return json({
|
|
objects: displayObjects.map((o) => ({
|
|
key: o.s3Key,
|
|
fileName: o.fileName,
|
|
mimeType: o.mimeType,
|
|
sizeBytes: o.sizeBytes,
|
|
fileType: o.fileType,
|
|
etag: o.fileHash,
|
|
lastModified: o.createdAt instanceof Date ? o.createdAt.toISOString() : new Date(o.createdAt).toISOString(),
|
|
downloadUrl: `${config.baseUrl}/f/${o.publicId}`,
|
|
})),
|
|
prefixes,
|
|
isTruncated,
|
|
nextContinuationToken: isTruncated ? displayObjects[displayObjects.length - 1]?.s3Key : null,
|
|
});
|
|
};
|
|
|
|
export const handleUploadObjectV1 = async (req: Request, params: RouteParams): Promise<Response> => {
|
|
const bucket = await findBucketByName(params.bucket!);
|
|
if (!bucket) return jsonError('Bucket not found', 404);
|
|
|
|
const formData = await req.formData();
|
|
const file = formData.get('file');
|
|
|
|
if (!file || !(file instanceof File)) {
|
|
return jsonError('No file provided', 400);
|
|
}
|
|
|
|
const key = (formData.get('key') as string) || file.name;
|
|
const buffer = Buffer.from(await file.arrayBuffer());
|
|
const hash = computeHash(buffer);
|
|
|
|
// Upload to Telegram
|
|
const tempPath = `/tmp/teleuploader-web-${nanoid()}`;
|
|
await Bun.write(tempPath, buffer);
|
|
|
|
const signatureBuffer = buffer.subarray(0, 16);
|
|
const { fileName: finalFileName, mimeType } = ensureExtension(key.split('/').pop() || 'file', signatureBuffer, file.type || 'application/octet-stream');
|
|
const fileType = 'document';
|
|
|
|
const forwardResult = await forwardToStorage(
|
|
createReadStream(tempPath),
|
|
`s3-${bucket.name}-${key.replace(/\//g, '_')}`,
|
|
fileType,
|
|
);
|
|
|
|
const publicId = nanoid();
|
|
const { db, files: fileSchema } = await import('../db/index');
|
|
|
|
await db.insert(fileSchema).values({
|
|
publicId,
|
|
telegramFileId: forwardResult.telegramFileId,
|
|
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
|
|
storageChatId: config.storageChatId,
|
|
storageMessageId: forwardResult.storageMessageId,
|
|
fileName: finalFileName,
|
|
mimeType,
|
|
sizeBytes: buffer.byteLength,
|
|
fileType,
|
|
uploaderId: 0,
|
|
fileHash: hash,
|
|
bucketId: bucket.id,
|
|
s3Key: key,
|
|
storageBackend: 'telegram',
|
|
isDeleted: false,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
|
|
await cleanupTempFile(tempPath);
|
|
|
|
return json({
|
|
key,
|
|
size: buffer.byteLength,
|
|
etag: hash,
|
|
downloadUrl: `${config.baseUrl}/f/${publicId}`,
|
|
}, 201);
|
|
};
|
|
|
|
export const handleDeleteObjectV1 = async (_req: Request, params: RouteParams): Promise<Response> => {
|
|
const bucket = await findBucketByName(params.bucket!);
|
|
if (!bucket) return jsonError('Bucket not found', 404);
|
|
await softDeleteFile(bucket.id, params.key!);
|
|
return json({ success: true });
|
|
};
|
|
|
|
export const handleDownloadObjectV1 = async (_req: Request, params: RouteParams): Promise<Response> => {
|
|
const bucket = await findBucketByName(params.bucket!);
|
|
if (!bucket) return jsonError('Bucket not found', 404);
|
|
|
|
const file = await findFileByBucketAndKey(bucket.id, params.key!);
|
|
if (!file) return jsonError('Object not found', 404);
|
|
|
|
const fileInfo = await getFileInfo(file.telegramFileId);
|
|
const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`;
|
|
|
|
return new Response(null, {
|
|
status: 302,
|
|
headers: { Location: redirectUrl },
|
|
});
|
|
};
|
|
|
|
export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Promise<Response> => {
|
|
const body = (await req.json()) as { sourceKey?: string; destBucket?: string; destKey?: string };
|
|
if (!body.sourceKey || !body.destKey) {
|
|
return jsonError('sourceKey and destKey are required', 400);
|
|
}
|
|
|
|
const destBucketName = body.destBucket || params.bucket!;
|
|
const sourceBucket = await findBucketByName(params.bucket!);
|
|
const destBucket = await findBucketByName(destBucketName);
|
|
|
|
if (!sourceBucket || !destBucket) return jsonError('Bucket not found', 404);
|
|
|
|
const sourceFile = await findFileByBucketAndKey(sourceBucket.id, body.sourceKey);
|
|
if (!sourceFile) return jsonError('Source object not found', 404);
|
|
|
|
const publicId = nanoid();
|
|
const { db, files: fileSchema } = await import('../db/index');
|
|
|
|
await db.insert(fileSchema).values({
|
|
publicId,
|
|
telegramFileId: sourceFile.telegramFileId,
|
|
telegramFileUniqueId: sourceFile.telegramFileUniqueId,
|
|
storageChatId: sourceFile.storageChatId,
|
|
storageMessageId: sourceFile.storageMessageId,
|
|
fileName: sourceFile.fileName,
|
|
mimeType: sourceFile.mimeType,
|
|
sizeBytes: sourceFile.sizeBytes,
|
|
fileType: sourceFile.fileType,
|
|
uploaderId: 0,
|
|
fileHash: sourceFile.fileHash,
|
|
bucketId: destBucket.id,
|
|
s3Key: body.destKey,
|
|
storageBackend: 'telegram',
|
|
isDeleted: false,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
|
|
return json({ sourceKey: body.sourceKey, destKey: body.destKey, destBucket: destBucketName });
|
|
};
|
|
|
|
// ─────── Router ───────
|
|
|
|
export const handleWebApiV1 = async (req: Request): Promise<Response> => {
|
|
const url = new URL(req.url);
|
|
const pathname = url.pathname.replace(/^\/api\/v1/, '');
|
|
const parts = pathname.split('/').filter(Boolean);
|
|
const method = req.method;
|
|
|
|
try {
|
|
// GET /api/v1/buckets
|
|
if (parts.length === 1 && parts[0] === 'buckets' && method === 'GET') {
|
|
return await handleListBucketsV1();
|
|
}
|
|
|
|
// POST /api/v1/buckets
|
|
if (parts.length === 1 && parts[0] === 'buckets' && method === 'POST') {
|
|
return await handleCreateBucketV1(req);
|
|
}
|
|
|
|
// DELETE /api/v1/buckets/{name}
|
|
if (parts.length === 2 && parts[0] === 'buckets' && method === 'DELETE') {
|
|
return await handleDeleteBucketV1(req, { bucket: parts[1] });
|
|
}
|
|
|
|
// GET /api/v1/buckets/{name}/objects
|
|
if (parts.length === 3 && parts[0] === 'buckets' && parts[2] === 'objects' && method === 'GET') {
|
|
return await handleListObjectsV1(req, { bucket: parts[1] });
|
|
}
|
|
|
|
// POST /api/v1/buckets/{name}/upload
|
|
if (parts.length === 3 && parts[0] === 'buckets' && parts[2] === 'upload' && method === 'POST') {
|
|
return await handleUploadObjectV1(req, { bucket: parts[1] });
|
|
}
|
|
|
|
// POST /api/v1/buckets/{name}/copy
|
|
if (parts.length === 3 && parts[0] === 'buckets' && parts[2] === 'copy' && method === 'POST') {
|
|
return await handleCopyObjectV1(req, { bucket: parts[1] });
|
|
}
|
|
|
|
// DELETE /api/v1/buckets/{name}/{key+}
|
|
if (parts.length >= 3 && parts[0] === 'buckets' && method === 'DELETE') {
|
|
const bucket = parts[1];
|
|
const key = parts.slice(2).join('/');
|
|
return await handleDeleteObjectV1(req, { bucket, key });
|
|
}
|
|
|
|
// GET /api/v1/buckets/{name}/download/{key+}
|
|
if (parts.length >= 4 && parts[0] === 'buckets' && parts[2] === 'download' && method === 'GET') {
|
|
const bucket = parts[1];
|
|
const key = parts.slice(3).join('/');
|
|
return await handleDownloadObjectV1(req, { bucket, key });
|
|
}
|
|
|
|
return jsonError('Not found', 404);
|
|
} catch (error: unknown) {
|
|
logger.error('Web API error', { path: pathname, error: getErrorMessage(error) });
|
|
return jsonError('Internal server error', 500);
|
|
}
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 2: Commit web API**
|
|
|
|
```bash
|
|
git add src/routes/web-api.ts
|
|
git commit -m "feat: add JSON v1 web API for S3 management UI"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Web File Manager UI
|
|
|
|
**Files:**
|
|
- Create: `src/home.html`
|
|
- Create: `src/routes/home.ts`
|
|
- Modify: `src/index.ts` (register new routes)
|
|
|
|
- [ ] **Step 1: Create `src/routes/home.ts`**
|
|
|
|
```typescript
|
|
import { config } from '../env';
|
|
|
|
export const handleHome = async (): Promise<Response> => {
|
|
const html = await Bun.file('src/home.html').text();
|
|
return new Response(html, {
|
|
status: 200,
|
|
headers: {
|
|
'content-type': 'text/html; charset=utf-8',
|
|
},
|
|
});
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 2: Create `src/home.html`**
|
|
|
|
This is a comprehensive single-page file manager UI. It must include:
|
|
- Bucket selector/create/delete
|
|
- Object listing with folder navigation
|
|
- Upload via drag & drop or file picker with progress bar
|
|
- Download and delete actions
|
|
- Copy link to clipboard
|
|
- Search/filter
|
|
- Dark/light mode
|
|
- Credentials display modal
|
|
|
|
The HTML is too long to reproduce in full here, but the core structure:
|
|
|
|
```html
|
|
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<title>TeleUploader · S3 File Manager</title>
|
|
<style>
|
|
:root {
|
|
--bg: #ffffff; --bg2: #f5f5f5; --text: #1a1a1a;
|
|
--text2: #666; --border: #e0e0e0; --accent: #2563eb;
|
|
--danger: #dc2626; --radius: 8px;
|
|
}
|
|
@media (prefers-color-scheme: dark) {
|
|
:root {
|
|
--bg: #0d1117; --bg2: #161b22; --text: #c9d1d9;
|
|
--text2: #8b949e; --border: #30363d; --accent: #58a6ff;
|
|
--danger: #f85149;
|
|
}
|
|
}
|
|
* { box-sizing: border-box; margin: 0; padding: 0; }
|
|
body {
|
|
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
|
background: var(--bg); color: var(--text); line-height: 1.5;
|
|
min-height: 100vh;
|
|
}
|
|
/* ─── Top Bar ─── */
|
|
.topbar {
|
|
display: flex; align-items: center; gap: 12px;
|
|
padding: 12px 24px; background: var(--bg2);
|
|
border-bottom: 1px solid var(--border);
|
|
position: sticky; top: 0; z-index: 50;
|
|
}
|
|
.topbar .logo { font-weight: 700; font-size: 1.1rem; }
|
|
.topbar select, .topbar button {
|
|
padding: 6px 12px; border: 1px solid var(--border);
|
|
border-radius: var(--radius); background: var(--bg);
|
|
color: var(--text); font-size: 0.875rem; cursor: pointer;
|
|
}
|
|
.topbar button.primary { background: var(--accent); color: #fff; border-color: var(--accent); }
|
|
.topbar .spacer { flex: 1; }
|
|
.topbar .search input {
|
|
padding: 6px 12px; border: 1px solid var(--border);
|
|
border-radius: var(--radius); background: var(--bg);
|
|
color: var(--text); font-size: 0.875rem; width: 200px;
|
|
}
|
|
/* ─── File List ─── */
|
|
.file-list { padding: 16px 24px; }
|
|
.breadcrumb {
|
|
padding: 8px 0; margin-bottom: 8px; font-size: 0.9rem;
|
|
color: var(--accent); cursor: pointer;
|
|
}
|
|
.breadcrumb span:hover { text-decoration: underline; }
|
|
.breadcrumb .sep { color: var(--text2); margin: 0 4px; }
|
|
.file-row {
|
|
display: flex; align-items: center; gap: 12px;
|
|
padding: 10px 12px; border-radius: var(--radius);
|
|
cursor: pointer; transition: background 0.1s;
|
|
}
|
|
.file-row:hover { background: var(--bg2); }
|
|
.file-row .icon { font-size: 1.2rem; width: 28px; text-align: center; flex-shrink: 0; }
|
|
.file-row .name { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
|
.file-row .size { width: 80px; text-align: right; color: var(--text2); font-size: 0.85rem; }
|
|
.file-row .date { width: 140px; color: var(--text2); font-size: 0.85rem; }
|
|
.file-row .actions { display: flex; gap: 4px; }
|
|
.file-row .actions button {
|
|
padding: 4px 8px; border: none; border-radius: 4px;
|
|
background: transparent; color: var(--text2); cursor: pointer;
|
|
font-size: 0.8rem;
|
|
}
|
|
.file-row .actions button:hover { color: var(--text); background: var(--border); }
|
|
/* ─── Drop Zone ─── */
|
|
.dropzone {
|
|
position: fixed; bottom: 0; left: 0; right: 0;
|
|
padding: 12px 24px; background: var(--bg2);
|
|
border-top: 1px solid var(--border);
|
|
text-align: center; color: var(--text2); font-size: 0.85rem;
|
|
cursor: pointer;
|
|
}
|
|
.dropzone.dragover {
|
|
background: var(--accent); color: #fff;
|
|
}
|
|
/* ─── Upload progress ─── */
|
|
.progress-overlay {
|
|
position: fixed; top: 0; left: 0; right: 0; bottom: 0;
|
|
background: rgba(0,0,0,0.5); display: flex;
|
|
align-items: center; justify-content: center; z-index: 100;
|
|
}
|
|
.progress-card {
|
|
background: var(--bg); padding: 24px; border-radius: var(--radius);
|
|
min-width: 300px; max-width: 500px;
|
|
}
|
|
.progress-bar {
|
|
height: 8px; background: var(--border); border-radius: 4px;
|
|
margin: 12px 0; overflow: hidden;
|
|
}
|
|
.progress-bar .fill {
|
|
height: 100%; background: var(--accent);
|
|
transition: width 0.2s; width: 0%;
|
|
}
|
|
/* ─── Modal ─── */
|
|
.modal-overlay {
|
|
position: fixed; top: 0; left: 0; right: 0; bottom: 0;
|
|
background: rgba(0,0,0,0.5); display: flex;
|
|
align-items: center; justify-content: center; z-index: 100;
|
|
}
|
|
.modal {
|
|
background: var(--bg); padding: 24px; border-radius: var(--radius);
|
|
min-width: 360px; max-width: 500px;
|
|
}
|
|
.modal h3 { margin-bottom: 16px; }
|
|
.modal input {
|
|
width: 100%; padding: 8px 12px; border: 1px solid var(--border);
|
|
border-radius: var(--radius); background: var(--bg);
|
|
color: var(--text); margin-bottom: 12px;
|
|
}
|
|
.modal .buttons { display: flex; gap: 8px; justify-content: flex-end; }
|
|
.modal .buttons button {
|
|
padding: 8px 16px; border: 1px solid var(--border);
|
|
border-radius: var(--radius); background: var(--bg);
|
|
color: var(--text); cursor: pointer;
|
|
}
|
|
.modal .buttons .primary { background: var(--accent); color: #fff; border-color: var(--accent); }
|
|
.modal .buttons .danger { background: var(--danger); color: #fff; border-color: var(--danger); }
|
|
/* ─── Empty state ─── */
|
|
.empty {
|
|
text-align: center; padding: 48px 24px; color: var(--text2);
|
|
}
|
|
.empty h2 { font-size: 1.2rem; margin-bottom: 8px; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="topbar">
|
|
<span class="logo">📦 TeleUploader</span>
|
|
<select id="bucketSelect" onchange="switchBucket(this.value)">
|
|
<option value="">— Select bucket —</option>
|
|
</select>
|
|
<button onclick="showCreateBucketModal()">+ New</button>
|
|
<button onclick="showCredentialsModal()" title="S3 Credentials">🔑</button>
|
|
<span class="spacer"></span>
|
|
<div class="search">
|
|
<input id="searchInput" type="text" placeholder="Filter prefix..." oninput="debouncedSearch()">
|
|
</div>
|
|
</div>
|
|
|
|
<div id="breadcrumb" class="breadcrumb" style="display:none;padding:8px 24px"></div>
|
|
|
|
<div id="fileList" class="file-list">
|
|
<div class="empty">
|
|
<h2>Select a bucket to get started</h2>
|
|
<p>Choose a bucket from the dropdown above, or create a new one.</p>
|
|
</div>
|
|
</div>
|
|
|
|
<div id="dropzone" class="dropzone" style="display:none">
|
|
📁 Drop files here or click to upload
|
|
</div>
|
|
|
|
<div id="progressOverlay" class="progress-overlay" style="display:none">
|
|
<div class="progress-card">
|
|
<h3>Uploading...</h3>
|
|
<div id="progressFileName"></div>
|
|
<div class="progress-bar"><div id="progressFill" class="fill"></div></div>
|
|
<div id="progressPercent" style="font-size:0.85rem;color:var(--text2)">0%</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div id="modalOverlay" class="modal-overlay" style="display:none" onclick="closeModal(event)">
|
|
<div id="modalContent" class="modal" onclick="event.stopPropagation()">
|
|
<!-- Dynamic modal content -->
|
|
</div>
|
|
</div>
|
|
|
|
<script>
|
|
// ─── State ───
|
|
let currentBucket = null;
|
|
let currentPrefix = '';
|
|
let currentObjects = [];
|
|
let currentPrefixes = [];
|
|
let allBuckets = [];
|
|
let searchTimer = null;
|
|
|
|
// ─── API helpers ───
|
|
const api = async (path, opts = {}) => {
|
|
const res = await fetch(path, opts);
|
|
if (!res.ok) {
|
|
const body = await res.json().catch(() => ({ error: res.statusText }));
|
|
throw new Error(body.error || res.statusText);
|
|
}
|
|
return res;
|
|
};
|
|
|
|
const apiJson = async (path, opts = {}) => {
|
|
const res = await api(path, { headers: { 'content-type': 'application/json' }, ...opts });
|
|
return res.json();
|
|
};
|
|
|
|
// ─── Load buckets ───
|
|
const loadBuckets = async () => {
|
|
const data = await apiJson('/api/v1/buckets');
|
|
allBuckets = data.buckets || [];
|
|
const sel = document.getElementById('bucketSelect');
|
|
sel.innerHTML = '<option value="">— Select bucket —</option>' +
|
|
allBuckets.map(b => `<option value="${b.name}">${b.name} (${b.objectCount})</option>`).join('');
|
|
if (currentBucket) sel.value = currentBucket;
|
|
};
|
|
|
|
// ─── Switch bucket ───
|
|
const switchBucket = async (name) => {
|
|
currentBucket = name || null;
|
|
currentPrefix = '';
|
|
if (name) {
|
|
await loadObjects();
|
|
document.getElementById('dropzone').style.display = 'block';
|
|
} else {
|
|
document.getElementById('fileList').innerHTML =
|
|
'<div class="empty"><h2>Select a bucket</h2><p>Choose a bucket from the dropdown above.</p></div>';
|
|
document.getElementById('breadcrumb').style.display = 'none';
|
|
document.getElementById('dropzone').style.display = 'none';
|
|
}
|
|
};
|
|
|
|
// ─── Breadcrumb ───
|
|
const renderBreadcrumb = () => {
|
|
const bc = document.getElementById('breadcrumb');
|
|
if (!currentPrefix) { bc.style.display = 'none'; return; }
|
|
bc.style.display = 'block';
|
|
const parts = currentPrefix.split('/').filter(Boolean);
|
|
bc.innerHTML = `<span onclick="navigateTo('')">${currentBucket}</span>`;
|
|
let accumulated = '';
|
|
for (const part of parts) {
|
|
accumulated += part + '/';
|
|
bc.innerHTML += `<span class="sep">/</span><span onclick="navigateTo('${accumulated}')">${part}</span>`;
|
|
}
|
|
};
|
|
|
|
const navigateTo = (prefix) => { currentPrefix = prefix; loadObjects(); };
|
|
|
|
// ─── Load objects ───
|
|
const loadObjects = async () => {
|
|
if (!currentBucket) return;
|
|
const searchVal = document.getElementById('searchInput').value;
|
|
const prefix = searchVal || currentPrefix;
|
|
const url = `/api/v1/buckets/${encodeURIComponent(currentBucket)}/objects?prefix=${encodeURIComponent(prefix)}&delimiter=/&max-keys=200`;
|
|
try {
|
|
const data = await apiJson(url);
|
|
currentObjects = data.objects || [];
|
|
currentPrefixes = data.prefixes || [];
|
|
renderFileList();
|
|
renderBreadcrumb();
|
|
} catch (e) {
|
|
document.getElementById('fileList').innerHTML =
|
|
`<div class="empty"><h2>Error</h2><p>${e.message}</p></div>`;
|
|
}
|
|
};
|
|
|
|
// ─── Render file list ───
|
|
const renderFileList = () => {
|
|
const container = document.getElementById('fileList');
|
|
if (currentPrefixes.length === 0 && currentObjects.length === 0) {
|
|
container.innerHTML = '<div class="empty"><h2>This bucket is empty</h2><p>Drop files here to upload.</p></div>';
|
|
return;
|
|
}
|
|
|
|
let html = '';
|
|
for (const prefix of currentPrefixes) {
|
|
const displayName = prefix.replace(currentPrefix, '');
|
|
html += `<div class="file-row" onclick="navigateTo('${prefix}')">
|
|
<span class="icon">🗂</span>
|
|
<span class="name">${displayName.endsWith('/') ? displayName : displayName + '/'}</span>
|
|
<span class="size">—</span>
|
|
<span class="date"></span>
|
|
<span class="actions"></span>
|
|
</div>`;
|
|
}
|
|
for (const obj of currentObjects) {
|
|
const displayName = obj.key.replace(currentPrefix, '');
|
|
const size = formatSize(obj.sizeBytes);
|
|
const date = formatDate(obj.lastModified);
|
|
html += `<div class="file-row">
|
|
<span class="icon">📄</span>
|
|
<span class="name">${escapeHtml(displayName)}</span>
|
|
<span class="size">${size}</span>
|
|
<span class="date">${date}</span>
|
|
<span class="actions">
|
|
<button onclick="event.stopPropagation();downloadObject('${obj.key}')" title="Download">⬇</button>
|
|
<button onclick="event.stopPropagation();copyLink('${obj.key}')" title="Copy link">🔗</button>
|
|
<button onclick="event.stopPropagation();deleteObject('${obj.key}')" title="Delete">🗑</button>
|
|
</span>
|
|
</div>`;
|
|
}
|
|
container.innerHTML = html;
|
|
};
|
|
|
|
// ─── Helpers ───
|
|
const formatSize = (bytes) => {
|
|
if (!bytes) return '0 B';
|
|
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
|
|
let i = 0; let size = bytes;
|
|
while (size >= 1024 && i < units.length - 1) { size /= 1024; i++; }
|
|
return `${size.toFixed(i > 0 ? 1 : 0)} ${units[i]}`;
|
|
};
|
|
|
|
const formatDate = (iso) => {
|
|
if (!iso) return '';
|
|
const d = new Date(iso);
|
|
return d.toLocaleDateString(undefined, { month: 'short', day: 'numeric', year: 'numeric' });
|
|
};
|
|
|
|
const escapeHtml = (s) => {
|
|
const div = document.createElement('div');
|
|
div.textContent = s;
|
|
return div.innerHTML;
|
|
};
|
|
|
|
const debouncedSearch = () => {
|
|
clearTimeout(searchTimer);
|
|
searchTimer = setTimeout(loadObjects, 300);
|
|
};
|
|
|
|
// ─── Actions ───
|
|
const downloadObject = async (key) => {
|
|
const url = `/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`;
|
|
window.open(url, '_blank');
|
|
};
|
|
|
|
const copyLink = (key) => {
|
|
const url = `${window.location.origin}/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`;
|
|
navigator.clipboard.writeText(url).catch(() => {});
|
|
};
|
|
|
|
const deleteObject = async (key) => {
|
|
if (!confirm(`Delete "${key}"?`)) return;
|
|
try {
|
|
await api(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/${encodeURIComponent(key)}`, { method: 'DELETE' });
|
|
await loadObjects();
|
|
} catch (e) {
|
|
alert(`Delete failed: ${e.message}`);
|
|
}
|
|
};
|
|
|
|
// ─── Upload ───
|
|
const uploadFiles = async (files) => {
|
|
if (!currentBucket || files.length === 0) return;
|
|
const overlay = document.getElementById('progressOverlay');
|
|
const progressFill = document.getElementById('progressFill');
|
|
const progressName = document.getElementById('progressFileName');
|
|
const progressPercent = document.getElementById('progressPercent');
|
|
overlay.style.display = 'flex';
|
|
|
|
for (let i = 0; i < files.length; i++) {
|
|
const file = files[i];
|
|
progressName.textContent = `${i + 1}/${files.length}: ${file.name}`;
|
|
progressFill.style.width = '0%';
|
|
progressPercent.textContent = '0%';
|
|
|
|
await new Promise((resolve, reject) => {
|
|
const formData = new FormData();
|
|
formData.append('file', file);
|
|
formData.append('key', currentPrefix + file.name);
|
|
|
|
const xhr = new XMLHttpRequest();
|
|
xhr.upload.onprogress = (e) => {
|
|
if (e.lengthComputable) {
|
|
const pct = Math.round((e.loaded / e.total) * 100);
|
|
progressFill.style.width = pct + '%';
|
|
progressPercent.textContent = pct + '%';
|
|
}
|
|
};
|
|
xhr.onload = () => {
|
|
if (xhr.status >= 200 && xhr.status < 300) resolve();
|
|
else reject(new Error(xhr.statusText));
|
|
};
|
|
xhr.onerror = () => reject(new Error('Upload failed'));
|
|
xhr.open('POST', `/api/v1/buckets/${encodeURIComponent(currentBucket)}/upload`);
|
|
xhr.send(formData);
|
|
});
|
|
}
|
|
|
|
overlay.style.display = 'none';
|
|
await loadObjects();
|
|
};
|
|
|
|
// ─── Drag & drop ───
|
|
const dropzone = document.getElementById('dropzone');
|
|
dropzone.addEventListener('dragover', (e) => { e.preventDefault(); dropzone.classList.add('dragover'); });
|
|
dropzone.addEventListener('dragleave', () => dropzone.classList.remove('dragover'));
|
|
dropzone.addEventListener('drop', (e) => {
|
|
e.preventDefault();
|
|
dropzone.classList.remove('dragover');
|
|
if (e.dataTransfer.files.length > 0) uploadFiles(e.dataTransfer.files);
|
|
});
|
|
dropzone.addEventListener('click', () => {
|
|
const input = document.createElement('input');
|
|
input.type = 'file';
|
|
input.multiple = true;
|
|
input.onchange = () => { if (input.files.length > 0) uploadFiles(input.files); };
|
|
input.click();
|
|
});
|
|
|
|
// ─── Modals ───
|
|
const showModal = (html) => {
|
|
document.getElementById('modalContent').innerHTML = html;
|
|
document.getElementById('modalOverlay').style.display = 'flex';
|
|
};
|
|
const closeModal = (e) => {
|
|
if (e && e.target !== e.currentTarget) return;
|
|
document.getElementById('modalOverlay').style.display = 'none';
|
|
};
|
|
|
|
const showCreateBucketModal = () => {
|
|
showModal(`
|
|
<h3>Create Bucket</h3>
|
|
<input id="bucketNameInput" type="text" placeholder="my-bucket-name" pattern="[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]">
|
|
<p style="font-size:0.8rem;color:var(--text2);margin-bottom:12px">Lowercase, 3-63 chars, no underscores</p>
|
|
<div class="buttons">
|
|
<button onclick="closeModal()">Cancel</button>
|
|
<button class="primary" onclick="createBucket()">Create</button>
|
|
</div>
|
|
`);
|
|
setTimeout(() => document.getElementById('bucketNameInput')?.focus(), 100);
|
|
};
|
|
|
|
const createBucket = async () => {
|
|
const name = document.getElementById('bucketNameInput').value.trim();
|
|
if (!name) return;
|
|
try {
|
|
await apiJson('/api/v1/buckets', { method: 'POST', body: JSON.stringify({ name }) });
|
|
closeModal();
|
|
await loadBuckets();
|
|
document.getElementById('bucketSelect').value = name;
|
|
await switchBucket(name);
|
|
} catch (e) {
|
|
alert(`Failed: ${e.message}`);
|
|
}
|
|
};
|
|
|
|
const showCredentialsModal = () => {
|
|
showModal(`
|
|
<h3>S3 Credentials</h3>
|
|
<p style="margin-bottom:12px;font-size:0.85rem;color:var(--text2)">
|
|
Use these credentials in any S3 client (aws-cli, rclone, s3cmd, etc.)
|
|
</p>
|
|
<label style="font-size:0.85rem;font-weight:600">Endpoint URL</label>
|
|
<input type="text" value="${window.location.origin}" readonly onclick="this.select()">
|
|
<label style="font-size:0.85rem;font-weight:600">Region</label>
|
|
<input type="text" value="us-east-1" readonly onclick="this.select()">
|
|
<label style="font-size:0.85rem;font-weight:600">Access Key</label>
|
|
<input id="s3AccessKey" type="text" readonly onclick="this.select()">
|
|
<label style="font-size:0.85rem;font-weight:600">Secret Key</label>
|
|
<input id="s3SecretKey" type="password" readonly onclick="this.select()">
|
|
<div class="buttons">
|
|
<button onclick="closeModal()">Close</button>
|
|
</div>
|
|
`);
|
|
};
|
|
|
|
// ─── Init ───
|
|
loadBuckets();
|
|
</script>
|
|
</body>
|
|
</html>
|
|
```
|
|
|
|
- [ ] **Step 3: Register routes in `src/index.ts`**
|
|
|
|
Add imports:
|
|
```typescript
|
|
import { handleHome } from './routes/home';
|
|
import { handleWebApiV1 } from './routes/web-api';
|
|
import { handleS3Request } from './routes/s3';
|
|
import { isS3Request } from './utils/s3/auth';
|
|
```
|
|
|
|
Add routes to the serve config (after existing routes, before server variable definition):
|
|
```typescript
|
|
serve({
|
|
// ... existing routes ...
|
|
'/': {
|
|
GET: handleHome,
|
|
},
|
|
// Web API v1
|
|
'/api/v1/*': {
|
|
GET: handleWebApiV1,
|
|
POST: handleWebApiV1,
|
|
DELETE: handleWebApiV1,
|
|
PUT: handleWebApiV1,
|
|
},
|
|
// S3-compatible API catch-all
|
|
$: async (req: Request) => {
|
|
// Only handle requests with SigV4 auth header
|
|
if (isS3Request(Object.fromEntries(req.headers))) {
|
|
return handleS3Request(req);
|
|
}
|
|
return new Response('Not Found', { status: 404 });
|
|
},
|
|
})
|
|
```
|
|
|
|
Note: Bun.serve() routes matching priority: literal paths first, then wildcard paths, then `$` catch-all. The `/api/v1/*` pattern is a wildcard route that matches any path starting with `/api/v1/`. For the S3 catch-all, we use `$` which is Bun's catch-all route that fires when no other route matches. Since the S3 catch-all is the last resort, we check the Authorization header to differentiate S3 from true 404s.
|
|
|
|
- [ ] **Step 4: Verify the app still compiles**
|
|
|
|
```bash
|
|
cd /mnt/code/TeleUploader && bun build src/index.ts --target=bun --outfile=dist/index.js
|
|
```
|
|
|
|
Expected: Build succeeds with no errors.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/routes/home.ts src/home.html src/index.ts
|
|
git commit -m "feat: add web file manager UI and S3 catch-all route"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: Tests
|
|
|
|
**Files:**
|
|
- Create: `test/s3-auth.test.ts`
|
|
- Create: `test/s3-operations.test.ts`
|
|
- Create: `test/web-api.test.ts`
|
|
- Modify: `package.json` (add test script entries)
|
|
|
|
- [ ] **Step 1: Create `test/s3-auth.test.ts`**
|
|
|
|
```typescript
|
|
import { describe, expect, it, beforeAll } from 'bun:test';
|
|
|
|
describe('S3 Auth (SigV4)', () => {
|
|
let verifySignature: typeof import('../src/utils/s3/auth').verifySignature;
|
|
let isS3Request: typeof import('../src/utils/s3/auth').isS3Request;
|
|
|
|
beforeAll(async () => {
|
|
const auth = await import('../src/utils/s3/auth');
|
|
verifySignature = auth.verifySignature;
|
|
isS3Request = auth.isS3Request;
|
|
});
|
|
|
|
it('should detect S3 requests by Authorization header', () => {
|
|
expect(isS3Request({ authorization: 'AWS4-HMAC-SHA256 Credential=...' })).toBe(true);
|
|
expect(isS3Request({ authorization: 'Bearer token123' })).toBe(false);
|
|
expect(isS3Request({})).toBe(false);
|
|
});
|
|
|
|
it('should reject missing Authorization header', async () => {
|
|
const result = await verifySignature('GET', 'http://localhost/', {}, null, 'key', 'secret', 'us-east-1');
|
|
expect(result.isValid).toBe(false);
|
|
});
|
|
|
|
it('should reject wrong access key', async () => {
|
|
const headers = {
|
|
authorization: 'AWS4-HMAC-SHA256 Credential=wrongkey/20260706/us-east-1/s3/aws4_request, SignedHeaders=host, Signature=abc123',
|
|
'x-amz-date': '20260706T120000Z',
|
|
};
|
|
const result = await verifySignature('GET', 'http://localhost/', headers, null, 'correctkey', 'secret', 'us-east-1');
|
|
expect(result.isValid).toBe(false);
|
|
expect(result.errorCode).toBe('SignatureDoesNotMatch');
|
|
});
|
|
|
|
it('should accept valid GET ListBuckets request', async () => {
|
|
// This tests that verification doesn't crash — full SigV4 signature
|
|
// construction requires exact header matching which is complex.
|
|
// The real signature verification is tested via integration with S3 clients.
|
|
const headers = {
|
|
authorization: 'AWS4-HMAC-SHA256 Credential=testkey/20260706/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000',
|
|
'x-amz-date': '20260706T120000Z',
|
|
'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
|
|
host: 'localhost',
|
|
};
|
|
const result = await verifySignature('GET', 'http://localhost/', headers, null, 'testkey', 'testsecret', 'us-east-1');
|
|
// Expected to not crash (the actual signature won't match with test values)
|
|
expect(result.isValid).toBe(false);
|
|
expect(result.errorCode).toBe('SignatureDoesNotMatch');
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Create `test/s3-operations.test.ts`**
|
|
|
|
Tests for S3 bucket and object operations. Because the S3 endpoints depend on Telegram (which is mocked), we test in isolation:
|
|
|
|
```typescript
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, it, mock } from 'bun:test';
|
|
|
|
// Mock DB and Telegram
|
|
const mockDbExecute = mock(() => Promise.resolve({ rows: [], rowCount: 0 }));
|
|
let mockSelectResult: unknown[] = [];
|
|
const mockDbSelect = mock(() => ({
|
|
from: () => ({
|
|
where: () => ({
|
|
orderBy: () => ({
|
|
limit: () => Promise.resolve(mockSelectResult),
|
|
}),
|
|
limit: () => Promise.resolve(mockSelectResult),
|
|
}),
|
|
}),
|
|
}));
|
|
|
|
mock.module('../src/db/index', () => ({
|
|
db: {
|
|
execute: mockDbExecute,
|
|
select: mockDbSelect,
|
|
insert: () => ({
|
|
values: () => Promise.resolve(),
|
|
}),
|
|
update: () => ({
|
|
set: () => ({
|
|
where: () => ({
|
|
returning: () => Promise.resolve([]),
|
|
}),
|
|
}),
|
|
}),
|
|
},
|
|
files: {},
|
|
}));
|
|
|
|
mock.module('../src/utils/telegram', () => ({
|
|
forwardToStorage: () => Promise.resolve({
|
|
telegramFileId: 'mock-tg-id',
|
|
telegramFileUniqueId: 'mock-tg-unique',
|
|
storageMessageId: 12345,
|
|
}),
|
|
getFileInfo: () => Promise.resolve({
|
|
file_size: 1000,
|
|
mime_type: 'application/octet-stream',
|
|
file_path: 'documents/file_0.dat',
|
|
bot_token: '123456:ABC-DEF',
|
|
}),
|
|
}));
|
|
|
|
mock.module('nanoid', () => ({
|
|
nanoid: () => 'mocked-nanoid-' + Math.random().toString(36).slice(2, 10),
|
|
}));
|
|
|
|
describe('S3 Bucket Operations', () => {
|
|
let s3Module: typeof import('../src/routes/s3');
|
|
let handleS3Request: typeof import('../src/routes/s3').handleS3Request;
|
|
|
|
beforeAll(async () => {
|
|
s3Module = await import('../src/routes/s3');
|
|
handleS3Request = s3Module.handleS3Request;
|
|
});
|
|
|
|
beforeEach(() => {
|
|
mockDbExecute.mockClear();
|
|
mockDbSelect.mockClear();
|
|
mockSelectResult = [];
|
|
process.env.S3_ACCESS_KEY = 'testkey';
|
|
process.env.S3_SECRET_KEY = 'testsecret';
|
|
process.env.S3_DEFAULT_REGION = 'us-east-1';
|
|
process.env.BOT_TOKEN = '123456:ABC-DEF';
|
|
process.env.STORAGE_CHANNEL_ID = '-1001234567890';
|
|
process.env.BASE_URL = 'http://localhost:4000';
|
|
process.env.DATABASE_URL = 'postgresql://localhost/test';
|
|
});
|
|
|
|
afterAll(() => {
|
|
mock.restore();
|
|
});
|
|
|
|
it('should return 403 for unauthorized requests', async () => {
|
|
const req = new Request('http://localhost:4000/', {
|
|
method: 'GET',
|
|
headers: { authorization: 'Invalid' },
|
|
});
|
|
const res = await handleS3Request(req);
|
|
expect(res.status).toBe(403);
|
|
const body = await res.text();
|
|
expect(body).toContain('AccessDenied');
|
|
expect(body).toContain('<?xml');
|
|
});
|
|
});
|
|
|
|
describe('S3 XML Builders', () => {
|
|
let xml: typeof import('../src/utils/s3/xml');
|
|
|
|
beforeAll(async () => {
|
|
xml = await import('../src/utils/s3/xml');
|
|
});
|
|
|
|
it('should build ListBuckets XML', () => {
|
|
const result = xml.listBucketsXml(
|
|
[{ name: 'test-bucket', createdAt: new Date('2026-01-01') }],
|
|
'req-1',
|
|
);
|
|
expect(result).toContain('<?xml');
|
|
expect(result).toContain('test-bucket');
|
|
expect(result).toContain('<ListAllMyBucketsResult');
|
|
expect(result).toContain('<CreationDate>');
|
|
});
|
|
|
|
it('should build ListBucketResult XML', () => {
|
|
const result = xml.listBucketResultXml(
|
|
'my-bucket',
|
|
[{ key: 'file.txt', sizeBytes: 100, etag: 'abc', lastModified: new Date(), mimeType: 'text/plain' }],
|
|
[],
|
|
false,
|
|
null,
|
|
1000,
|
|
'',
|
|
null,
|
|
null,
|
|
'req-1',
|
|
);
|
|
expect(result).toContain('file.txt');
|
|
expect(result).toContain('<ListBucketResult');
|
|
expect(result).toContain('<Size>100</Size>');
|
|
});
|
|
|
|
it('should build ListBucketV2 XML', () => {
|
|
const result = xml.listBucketV2ResultXml(
|
|
'my-bucket',
|
|
[{ key: 'a.txt', sizeBytes: 50, etag: 'def', lastModified: new Date(), mimeType: 'text/plain' }],
|
|
['photos/'],
|
|
false,
|
|
1000,
|
|
'',
|
|
'/',
|
|
null,
|
|
null,
|
|
1,
|
|
'req-2',
|
|
);
|
|
expect(result).toContain('<ListBucketResultV2');
|
|
expect(result).toContain('a.txt');
|
|
expect(result).toContain('<Prefix>photos/</Prefix>');
|
|
});
|
|
|
|
it('should build InitiateMultipartUpload XML', () => {
|
|
const result = xml.initiateMultipartUploadXml('bucket', 'key', 'upload-123');
|
|
expect(result).toContain('<UploadId>upload-123</UploadId>');
|
|
});
|
|
|
|
it('should build CopyObjectResult XML', () => {
|
|
const result = xml.copyObjectResultXml('etag-abc', new Date());
|
|
expect(result).toContain('<CopyObjectResult');
|
|
expect(result).toContain('etag-abc');
|
|
});
|
|
|
|
it('should build error XML', () => {
|
|
const result = xml.s3ErrorXml('NoSuchBucket', 'The specified bucket does not exist', '/bucket', 'req-1');
|
|
expect(result).toContain('<Code>NoSuchBucket</Code>');
|
|
expect(result).toContain('<Error>');
|
|
});
|
|
|
|
it('should parse DeleteObjects body', () => {
|
|
const body = `<Delete><Object><Key>file1.txt</Key></Object><Object><Key>file2.txt</Key></Object></Delete>`;
|
|
const { keys } = xml.parseDeleteObjectsBody(body);
|
|
expect(keys).toEqual(['file1.txt', 'file2.txt']);
|
|
});
|
|
|
|
it('should parse CompleteMultipartUpload body', () => {
|
|
const body = `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"abc"</ETag></Part><Part><PartNumber>2</PartNumber><ETag>"def"</ETag></Part></CompleteMultipartUpload>`;
|
|
const parts = xml.parseCompleteMultipartBody(body);
|
|
expect(parts).toHaveLength(2);
|
|
expect(parts[0]).toEqual({ partNumber: 1, etag: 'abc' });
|
|
expect(parts[1]).toEqual({ partNumber: 2, etag: 'def' });
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 3: Create `test/web-api.test.ts`**
|
|
|
|
```typescript
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, it, mock } from 'bun:test';
|
|
|
|
// Mock DB layer
|
|
const mockBuckets = [
|
|
{ id: 'uuid-1', name: 'test-bucket', createdAt: new Date(), updatedAt: new Date() },
|
|
];
|
|
|
|
const mockDbExecute = mock((sql: unknown) => {
|
|
const sqlStr = String(sql);
|
|
if (sqlStr.includes('FROM buckets WHERE name =')) {
|
|
return Promise.resolve({ rows: mockBuckets.filter(b => sqlStr.includes(b.name)), rowCount: 1 });
|
|
}
|
|
if (sqlStr.includes('FROM buckets ORDER BY')) {
|
|
return Promise.resolve({ rows: mockBuckets, rowCount: mockBuckets.length });
|
|
}
|
|
return Promise.resolve({ rows: [], rowCount: 0 });
|
|
});
|
|
|
|
mock.module('../src/db/index', () => ({
|
|
db: {
|
|
execute: mockDbExecute,
|
|
select: () => ({
|
|
from: () => ({
|
|
where: () => ({
|
|
limit: () => Promise.resolve([]),
|
|
}),
|
|
}),
|
|
}),
|
|
},
|
|
files: {},
|
|
}));
|
|
|
|
mock.module('../src/utils/telegram', () => ({
|
|
forwardToStorage: () => Promise.resolve({
|
|
telegramFileId: 'mock-tg-id',
|
|
telegramFileUniqueId: 'mock-tg-unique',
|
|
storageMessageId: 12345,
|
|
}),
|
|
getFileInfo: () => Promise.resolve({
|
|
file_size: 100,
|
|
mime_type: 'text/plain',
|
|
file_path: 'documents/file.txt',
|
|
bot_token: '123456:ABC-DEF',
|
|
}),
|
|
}));
|
|
|
|
mock.module('nanoid', () => ({
|
|
nanoid: () => 'mocked-nanoid-' + Math.random().toString(36).slice(2, 10),
|
|
}));
|
|
|
|
describe('Web API v1', () => {
|
|
let handleWebApiV1: typeof import('../src/routes/web-api').handleWebApiV1;
|
|
|
|
beforeAll(async () => {
|
|
const webApi = await import('../src/routes/web-api');
|
|
handleWebApiV1 = webApi.handleWebApiV1;
|
|
});
|
|
|
|
beforeEach(() => {
|
|
mockDbExecute.mockClear();
|
|
process.env.BOT_TOKEN = '123456:ABC-DEF';
|
|
process.env.STORAGE_CHANNEL_ID = '-1001234567890';
|
|
process.env.BASE_URL = 'http://localhost:4000';
|
|
process.env.DATABASE_URL = 'postgresql://localhost/test';
|
|
});
|
|
|
|
afterAll(() => {
|
|
mock.restore();
|
|
});
|
|
|
|
it('should list buckets via GET /api/v1/buckets', async () => {
|
|
const req = new Request('http://localhost:4000/api/v1/buckets');
|
|
const res = await handleWebApiV1(req);
|
|
expect(res.status).toBe(200);
|
|
const data = await res.json();
|
|
expect(data).toHaveProperty('buckets');
|
|
expect(Array.isArray(data.buckets)).toBe(true);
|
|
});
|
|
|
|
it('should return 404 for unknown API path', async () => {
|
|
const req = new Request('http://localhost:4000/api/v1/unknown');
|
|
const res = await handleWebApiV1(req);
|
|
expect(res.status).toBe(404);
|
|
const data = await res.json();
|
|
expect(data).toHaveProperty('error');
|
|
});
|
|
|
|
it('should return bucket object listing', async () => {
|
|
const req = new Request('http://localhost:4000/api/v1/buckets/test-bucket/objects?prefix=');
|
|
const res = await handleWebApiV1(req);
|
|
// Should return 200 even with empty results
|
|
expect(res.status).toBe(200);
|
|
const data = await res.json();
|
|
expect(data).toHaveProperty('objects');
|
|
expect(data).toHaveProperty('prefixes');
|
|
});
|
|
|
|
it('should reject invalid bucket name on create', async () => {
|
|
const req = new Request('http://localhost:4000/api/v1/buckets', {
|
|
method: 'POST',
|
|
headers: { 'content-type': 'application/json' },
|
|
body: JSON.stringify({ name: 'INVALID_NAME!' }),
|
|
});
|
|
const res = await handleWebApiV1(req);
|
|
expect(res.status).toBe(400);
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 4: Add test scripts to `package.json`**
|
|
|
|
Add these scripts:
|
|
```json
|
|
"test:s3-auth": "bun test test/s3-auth.test.ts",
|
|
"test:s3-ops": "bun test test/s3-operations.test.ts",
|
|
"test:web-api": "bun test test/web-api.test.ts",
|
|
"test:s3": "bun test test/s3-auth.test.ts && bun test test/s3-operations.test.ts && bun test test/web-api.test.ts"
|
|
```
|
|
|
|
- [ ] **Step 5: Run all tests**
|
|
|
|
```bash
|
|
cd /mnt/code/TeleUploader && bun test
|
|
```
|
|
|
|
Expected: All existing tests pass + new S3 tests pass.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add test/s3-auth.test.ts test/s3-operations.test.ts test/web-api.test.ts package.json
|
|
git commit -m "test: add S3 auth, operations, and web API tests"
|
|
```
|
|
|
|
---
|
|
|
|
## Spec Self-Review Check
|
|
|
|
- ✅ **Spec coverage**: Every spec section has corresponding tasks:
|
|
- DB schema → Task 1
|
|
- DB CRUD layers → Task 2
|
|
- S3 Auth + XML → Task 3
|
|
- S3 bucket/object/listing/multipart handlers → Task 4
|
|
- JSON web API → Task 5
|
|
- Web file manager UI → Task 6
|
|
- Tests → Task 7
|
|
- ✅ **No placeholders**: All steps contain actual code
|
|
- ✅ **Type consistency**: Function names used across tasks match (e.g., `createBucket` in Task 2 is consumed by `handleCreateBucket` in Task 4)
|
|
|
|
## Execution
|
|
|
|
Plan complete and saved to `docs/superpowers/plans/2026-07-06-s3-compatible-teleuploader.md`.
|
|
|
|
**Two execution options:**
|
|
|
|
1. **Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration
|
|
|
|
2. **Inline Execution** — Execute tasks in this session using executing-plans, batch execution with checkpoints
|
|
|
|
**Which approach?**
|