From 9c12d135e402fba1c4d94d2a194b86e9ee07ba5e Mon Sep 17 00:00:00 2001 From: asepharyana Date: Fri, 24 Jul 2026 10:18:45 +0700 Subject: [PATCH] Implement code changes to enhance functionality and improve performance --- docs/plan/tools/implementation.md | 1899 +++++++++++++++++++++++++---- 1 file changed, 1634 insertions(+), 265 deletions(-) diff --git a/docs/plan/tools/implementation.md b/docs/plan/tools/implementation.md index 91447ea..43af0e9 100644 --- a/docs/plan/tools/implementation.md +++ b/docs/plan/tools/implementation.md @@ -1,322 +1,1691 @@ -# Implementation Plan +# Implementation Plan — Granular Task Breakdown -## Phase Breakdown +Setiap task adalah unit kerja terkecil yang bisa dikerjakan dalam 1-4 jam. Format: ``` -Phase 1 — Foundation + Document Scanner MVP -├── Milestone 1.1: Rust backend skeleton (Gateway + Worker + NATS + Redis) -├── Milestone 1.2: Scanner pipeline core (edge → warp → enhance → B&W) -├── Milestone 1.3: Next.js frontend + upload + download -└── Milestone 1.4: OCR + searchable PDF + WebSocket progress - -Phase 2 — Scanner Complete + Image Tools -├── Milestone 2.1: Scanner fallback manual crop + PWA camera -├── Milestone 2.2: Image compress, resize, convert (WASM client-side) -├── Milestone 2.3: Background removal (ONNX) -└── Milestone 2.4: Batch processing - -Phase 3 — PDF Tools + Video/Audio -├── Milestone 3.1: PDF merge, split, images-to-pdf -├── Milestone 3.2: Video compress, extract audio -└── Milestone 3.3: Final polish + load testing +[ID] Task description + Files: path/to/file + Accept: criteria yang harus terpenuhi ``` --- -## Phase 1: Foundation + Document Scanner MVP +## Phase 1: Foundation + Document Scanner (Prioritas) ### Milestone 1.1 — Rust Backend Skeleton -**Goal**: Gateway + Worker bisa connected ke NATS + Redis, upload flow work. +**Goal**: Gateway + Worker connected ke NATS + Redis, upload flow end-to-end. -**Tasks**: +#### 1.1.1 — Init Rust Workspace -| # | Task | Files | Detail | -|---|------|-------|--------| -| 1.1.1 | Init Rust workspace | `apps/tools/backend/Cargo.toml` | Workspace dengan 3 crate: `gateway`, `workers`, `common` | -| 1.1.2 | Common types | `common/src/types.rs` | `JobStatus`, `Job`, `Tool`, `ScanOptions`, `UploadResponse` | -| 1.1.3 | Common error | `common/src/error.rs` | `PipelineError`, `UploadError`, `NatsError` | -| 1.1.4 | NATS subjects | `common/src/nats.rs` | Constants untuk semua subject/stream | -| 1.1.5 | Gateway: config | `gateway/src/config.rs` | Env-based config (Redis URL, NATS URL, storage path) | -| 1.1.6 | Gateway: upload route | `gateway/src/routes/upload.rs` | Multipart upload, validasi, save ke temp, publish NATS | -| 1.1.7 | Gateway: job status | `gateway/src/routes/job.rs` | GET job status dari Redis | -| 1.1.8 | Gateway: download | `gateway/src/routes/download.rs` | Stream file dari storage | -| 1.1.9 | Gateway: NATS publish | `gateway/src/nats/publisher.rs` | Publish job + progress | -| 1.1.10 | Gateway: Redis job | `gateway/src/redis/job.rs` | CRUD job status di Redis | -| 1.1.11 | Gateway: main | `gateway/src/main.rs` | Axum app bootstrap + routes | -| 1.1.12 | Workers: main loop | `workers/src/main.rs` | NATS consumer, dispatch ke tool handler | -| 1.1.13 | Workers: NATS consumer | `workers/src/nats/consumer.rs` | Subscribe jobs queue, ack/nack | -| 1.1.14 | Worker: scanner stub | `workers/src/scanner/mod.rs` | Cuma menerima job, update progress, complete | +``` +[1.1.1] Buat Cargo workspace dengan 4 crate: common, gateway, workers, wasm + Files: + apps/tools/backend/Cargo.toml (workspace definition) + apps/tools/backend/common/Cargo.toml (serde, uuid, chrono) + apps/tools/backend/gateway/Cargo.toml (axum, tokio, tower, async-nats, deadpool-redis, redis) + apps/tools/backend/workers/Cargo.toml (tokio, async-nats, redis, image, imageproc, lopdf, leptess, ort, rayon) + apps/tools/backend/wasm/Cargo.toml (wasm-bindgen, image, console-error-panic) + apps/tools/backend/rust-toolchain.toml (channel = "1.85") + Accept: cargo build —release works untuk semua crate (walaupun main.rs masih empty) +``` -**Acceptance**: `curl -X POST -F "file=@test.jpg" -F "tool=scan" http://localhost:3001/api/upload` → return job_id, setelah beberapa detik `GET /api/job/:id` return completed. +#### 1.1.2 — Common Types -**Effort**: ~3-4 hari +``` +[1.1.2] Define shared types: JobStatus enum, Job struct, Tool enum, ScanOptions, ImageOptions, + PdfOptions, UploadResponse, JobProgress + Files: + apps/tools/backend/common/src/lib.rs + apps/tools/backend/common/src/types.rs + Types: + JobStatus { Queued, Processing { stage: String, progress: u8 }, Completed, Failed(String) } + Tool { Scan, ImageCompress, ImageResize, ImageConvert, RemoveBg, + PdfMerge, PdfSplit, ImagesToPdf, PdfCompress, PdfToImages, + VideoCompress, AudioExtract, VideoTrim, GifMaker, AudioConvert } + Job { id: Uuid, tool: Tool, status: JobStatus, file_path: PathBuf, + result_path: Option, file_size: u64, options: Value, + created_at: DateTime, ttl_seconds: u64 } + JobProgress { job_id: Uuid, status: JobStatus, stage: String, progress: u8, message: String } + Accept: Semua type implements Serialize + Deserialize + Debug + Clone +``` + +#### 1.1.3 — Common Errors + +``` +[1.1.3] Define error types dengan thiserror + Files: + apps/tools/backend/common/src/error.rs + Errors: + UploadError (InvalidMime, FileTooLarge, Io, VirusDetected) + PipelineError (ImageLoad, EdgeDetection, Warp, Ocr, PdfGeneration, Timeout, Internal) + NatsError (Publish, Subscribe, JetStream, Timeout) + RedisError (Connection, Query, Serialization) + Accept: Setiap error punya Display + Source chain yang jelas +``` + +#### 1.1.4 — NATS Subjects & Streams + +``` +[1.1.4] Define NATS subject constants + stream configuration + Files: + apps/tools/backend/common/src/nats.rs + Subjects: + tools.scan.jobs → Queue for scan workers + tools.scan.progress → Fan-out progress events + tools.image.jobs + tools.image.progress + tools.pdf.jobs + tools.pdf.progress + tools.video.jobs + tools.video.progress + tools.scheduler.cleanup → Cron: cleanup expired files + Streams: + tools-jobs (max_age: 24h, storage: file) + tools-progress (max_age: 1h, storage: memory) + Accept: Unit test verifikasi format subject string +``` + +#### 1.1.5 — Gateway Config + +``` +[1.1.5] Environment-based configuration loader + Files: + apps/tools/backend/gateway/src/config.rs + Env vars: + GATEWAY_PORT (default: 3001) + NATS_URL (default: nats://localhost:4222) + REDIS_URL (default: redis://localhost:6379) + STORAGE_PATH (default: /data/tools) + MAX_FILE_SIZE_MB (default: 50) + JOB_TTL_SECONDS (default: 3600) + RATE_LIMIT_PER_MINUTE (default: 30) + RUST_LOG (default: info) + Accept: AppConfig struct dengan semua field, load dari env + fallback default +``` + +#### 1.1.6 — Gateway: Upload Route (POST /api/upload) + +``` +[1.1.6] Multipart file upload handler + Files: + apps/tools/backend/gateway/src/routes/upload.rs + apps/tools/backend/gateway/src/routes/mod.rs + Logic: + 1. Extract multipart: file + tool + options + 2. Validate MIME type (image/* for scan/image tools, application/pdf for pdf tools, + video/* for video tools, audio/* for audio tools) + 3. Check file size < MAX_FILE_SIZE_MB + 4. Scan magic bytes: verify actual content matches extension + 5. Save file ke {STORAGE_PATH}/upload/{uuid}.{ext} + 6. Generate job_id (Uuid v4) + 7. Save job metadata ke Redis: SET job:{job_id} → JSON + 8. Publish ke NATS: tools.{tool}.jobs → {job_id, file_path, options} + 9. Return 202: { job_id, status: "queued", tool, ws_url } + Accept: curl upload → 202 + job_id. curl job_status → status field populated +``` + +#### 1.1.7 — Gateway: Job Status Route (GET /api/job/{id}) + +``` +[1.1.7] Job status and metadata retrieval + Files: + apps/tools/backend/gateway/src/routes/job.rs + Logic: + 1. Extract job_id from path + 2. GET job:{job_id} from Redis + 3. Return 404 kalau not found + 4. Return JSON: { job_id, status, tool, progress, stage, message, + result: Option<{ download_url, file_size, file_name, preview_url }>, + created_at, error: Option } + Endpoints: + GET /api/job/{id} → Single job status + GET /api/job/{id}/preview → Preview image (thumbnail) + Accept: curl GET → full job status JSON +``` + +#### 1.1.8 — Gateway: Download Route (GET /api/download/{id}) + +``` +[1.1.8] File download with streaming + auto-cleanup awareness + Files: + apps/tools/backend/gateway/src/routes/download.rs + Logic: + 1. Extract job_id, get status from Redis + 2. Check status == completed → get result_path + 3. Stream file via tokio::fs::File → axum body stream + 4. Set Content-Disposition header with original filename + 5. Set Content-Type based on file extension + 6. Handle 404 (not found), 400 (not completed yet), 410 (expired) + Accept: curl GET /api/download/{id} → file download dengan correct headers +``` + +#### 1.1.9 — Gateway: Health & Metrics + +``` +[1.1.9] Health check + Prometheus metrics endpoint + Files: + apps/tools/backend/gateway/src/routes/health.rs + apps/tools/backend/gateway/src/metrics.rs + Endpoints: + GET /health → 200 OK (used by Traefik health check) + GET /metrics → Prometheus text format + Metrics (via custom counters, no external crate): + tools_uploaded_files_total → counter, labels: tool, status + tools_jobs_total → counter, labels: tool, status + tools_processing_duration_ms → histogram, labels: tool + tools_queue_depth → gauge, labels: tool + Accept: /health returns 200, /metrics returns prometheus-format text +``` + +#### 1.1.10 — Gateway: NATS Publisher + +``` +[1.1.10] NATS connection management + publish helpers + Files: + apps/tools/backend/gateway/src/nats/mod.rs + apps/tools/backend/gateway/src/nats/publisher.rs + Functions: + connect_nats(url: &str) → Result + publish_job(nats: &Connection, tool: Tool, payload: &Job) → Result<()> + publish_progress(nats: &Connection, progress: &JobProgress) → Result<()> + Accept: Integration test: publish message → consume back via subscriber +``` + +#### 1.1.11 — Gateway: Redis Client + +``` +[1.1.11] Redis connection pool + job CRUD operations + Files: + apps/tools/backend/gateway/src/redis/mod.rs + apps/tools/backend/gateway/src/redis/job.rs + apps/tools/backend/gateway/src/redis/ratelimit.rs + Functions: + connect_redis(url: &str) → Result + job_create(redis, job) → Result<()> + job_get(redis, job_id) → Result + job_update(redis, job_id, status) → Result<()> + job_delete(redis, job_id) → Result<()> + rate_limit_check(redis, ip, tool) → Result // sliding window + TTL: Set TTL JOB_TTL_SECONDS on job_create + Accept: Redis integration test: create → get → update → delete → not found +``` + +#### 1.1.12 — Gateway: Main Bootstrap + +``` +[1.1.12] Axum app assembly + graceful shutdown + Files: + apps/tools/backend/gateway/src/main.rs + Logic: + 1. Load config + 2. Init Redis connection pool + 3. Init NATS connection + 4. Build Axum router with all routes + 5. Spawn NATS progress consumer (subscribe tools.*.progress, caches in Redis) + 6. Start HTTP server on configured port + 7. Graceful shutdown on SIGINT/SIGTERM + States: SharedState { redis, nats, config } wrapped in Arc + Accept: cargo run —bin gateway → server listening on :3001 +``` + +#### 1.1.13 — Workers: NATS Consumer + +``` +[1.1.13] NATS JetStream consumer for job queues + Files: + apps/tools/backend/workers/src/main.rs + apps/tools/backend/workers/src/nats/mod.rs + apps/tools/backend/workers/src/nats/consumer.rs + Logic: + 1. Connect to NATS + Redis + 2. Subscribe to JetStream streams: tools.*.jobs + 3. For each message: + a. Deserialize job payload + b. Match tool → dispatch to appropriate handler + c. Acknowledge after handler returns + d. NACK with delay on failure (for retry) + 4. Max delivery: 3, then dead-letter + Max concurrency: configurable (TOOLS_WORKER_CONCURRENCY, default: 4) + Accept: Workers start, consume NATS messages, dispatch to tool handlers +``` + +#### 1.1.14 — Workers: Progress Publisher + +``` +[1.1.14] Progress reporting from worker to NATS → Gateway → WebSocket + Files: + apps/tools/backend/workers/src/nats/progress.rs + Functions: + report_progress(redis, nats, job_id, status, stage, progress, message) + 1. Update Redis: job:{job_id} status + progress + 2. Publish to NATS: tools.{tool}.progress → {job_id, status, stage, progress, message} + Helper: ProgressReporter struct yang implements Clone, bisa dipass ke pipeline + Accept: Worker updates progress → Gateway receives → WebSocket forwards +``` + +#### 1.1.15 — Worker: Scanner Stub + +``` +[1.1.15] Scanner worker yang bisa menerima job dan update progress + Files: + apps/tools/backend/workers/src/scanner/mod.rs + Logic: + 1. Receive job: { job_id, file_path, options } + 2. report_progress(queued → processing:10%) + 3. sleep 2s (simulasi pipeline) + 4. report_progress(processing:50%) + 5. sleep 2s + 6. Copy input file to output (simulasi hasil) + 7. report_progress(completed:100%) + 8. Update Redis with result_path + Accept: Upload → NATS queue → worker consume → progress update → completed +``` --- ### Milestone 1.2 — Scanner Pipeline Core -**Goal**: Image processing pipeline bisa ngubah foto miring jadi lurus + bersih + hitam-putih. Belum termasuk OCR dan PDF. +**Goal**: Foto miring → lurus + bersih + hitam-putih (belum OCR/PDF). -**Tasks**: +#### 1.2.1 — Preprocess: Load & Resize -| # | Task | Detail | Referensi | -|---|------|--------|-----------| -| 1.2.1 | Edge detection | Canny + morphological close + adaptive fallback | `pipeline.md` Stage 2-3 | -| 1.2.2 | Corner detection | Contour detection, largest rectangle filter, polygon approximation | `pipeline.md` Stage 3 | -| 1.2.3 | Perspective warp | DLT homography + backward mapping + bilinear interpolation | `pipeline.md` Stage 4 | -| 1.2.4 | Shadow removal | Adaptive illumination correction + Retinex | `pipeline.md` Stage 5 | -| 1.2.5 | Binarization | Sauvola local threshold + integral image optimization | `pipeline.md` Stage 6 | -| 1.2.6 | Deskew | Hough transform line detection + rotation | `pipeline.md` Stage 7 | -| 1.2.7 | Image enhancement | CLAHE + sharpen + contrast | `pipeline.md` Stage 5 | -| 1.2.8 | Pipeline assembly | All stages connected, progress callback per stage | `pipeline.md` Complete assembly | - -**Critical Algorithm**: Perspective warp via SVD untuk homography matrix. Butuh implementasi DLT algorithm atau pin `nalgebra` crate. - -```rust -// Pseudo untuk testing before optimization -// imageproc contour → corner detection → warp -let edges = robust_edge_detection(&gray); -let corners = find_document_corners(&edges)?; -let warped = perspective_warp(&original, corners); -let cleaned = remove_shadow(&warped); -let binary = binarize(&cleaned); -let final_img = deskew(&binary); +``` +[1.2.1] Load image from file, resize if too large, convert to grayscale + Files: + apps/tools/backend/workers/src/scanner/preprocess.rs + Functions: + load_image(path: &Path) → Result + safe_resize(img: DynamicImage, max_dim: u32) → DynamicImage + to_grayscale(img: &DynamicImage) → GrayImage + Rules: + - Resize if max(width, height) > 2000px → scale down, preserve aspect ratio + - Lanczos3 filter untuk downscale (sharpest) + - Support input: JPEG, PNG, WebP, HEIC (if feature enabled) + Accept: Unit test: 12MP image → resize to ≤2000px, verify aspect ratio preserved ``` -**Acceptance**: Image foto miring test → keluar hasil lurus bersih hitam-putih. +#### 1.2.2 — Edge Detection: Canny + Morphological Close -**Effort**: ~5-7 hari (ini bagian paling susah) +``` +[1.2.2] Canny edge detection with morphological operations to connect broken edges + Files: + apps/tools/backend/workers/src/scanner/edge.rs + Functions: + detect_edges(img: &GrayImage) → GrayImage + morphological_close(edges: &GrayImage, kernel_size: u8) → GrayImage + Algorithm: + 1. Gaussian blur (sigma=1.0) on grayscale + 2. Canny with low_threshold=50, high_threshold=150 + 3. Morphological close: dilate → erode with 5x5 kernel + 4. If edge_count < 1% of total pixels → retry Canny(20, 80) + Accept: Unit test: known test images → edge image with continuous document borders +``` + +#### 1.2.3 — Corner Detection: Largest Rectangle Contour + +``` +[1.2.3] Find 4 corners of the document from edge image + Files: + apps/tools/backend/workers/src/scanner/corners.rs + Functions: + find_contours(edges: &GrayImage) → Vec + largest_rectangular_contour(contours: &[Contour]) -> Option + approx_polygon(contour: &Contour, num_vertices: u32) → Option> + order_corners(points: Vec<(f64,f64)>) -> [(f64,f64); 4] + detect_corners(img: &GrayImage) -> Result<[(f64,f64); 4], FallbackReason> + Algorithm: + 1. imageproc::contours::find_contours + 2. Filter by area > 20% of total image + 3. Top 5 largest by contour area + 4. For each: approx polygon, find 4-vertex polygon + 5. Order: top-left, top-right, bottom-right, bottom-left + Accept: Unit test: 5 test images (normal, dark, angle, shadow, cluttered bg) + → correct corners or explicit fallback +``` + +#### 1.2.4 — Perspective Warp: DLT Homography + +``` +[1.2.4] Compute homography matrix via DLT + SVD, apply perspective warp + Files: + apps/tools/backend/workers/src/scanner/warp.rs + Functions: + compute_homography(src: &[(f64,f64);4], dst: &[(f64,f64);4]) → [[f64;3];3] + invert_homography(h: &[[f64;3];3]) → [[f64;3];3] + apply_homography(h: &[[f64;3];3], x: f64, y: f64) → (f64, f64) + bilinear_interpolate(img: &GrayImage, x: f64, y: f64) → Luma + warp_perspective(img: &DynamicImage, corners: [(f64,f64);4]) → DynamicImage + Algorithm: + DLT (Direct Linear Transform): + - 4 point correspondences → 8x9 matrix A + - SVD (via ndarray + ndarray-linalg or nalgebra) + - H = last column of V, reshape to 3x3 + Backward mapping: + - For each output pixel (x,y), compute source (sx,sy) via H_inv + - Bilinear interpolate from source + Accept: Unit test: 4 corners of known grid → warped image is perfectly rectangular +``` + +#### 1.2.5 — Shadow Removal: Illumination Correction + +``` +[1.2.5] Remove uneven lighting, shadows, and glare + Files: + apps/tools/backend/workers/src/scanner/shadow.rs + Functions: + gaussian_blur_large(img: &GrayImage, radius: f64) → GrayImage + subtract_background(img: &GrayImage, background: &GrayImage) -> GrayImage + apply_clahe(img: &GrayImage, tile_size: u8, clip_limit: u8) -> GrayImage + remove_shadow(img: &GrayImage) → GrayImage + Algorithm (primary): + 1. Large Gaussian blur (radius = max_dim/50, min 15px) = illumination estimate + 2. Subtract: pixel = max(0, original - background + mean(background)) + 3. CLAHE: 8x8 tiles, clip limit 3 + Algorithm (fallback - Retinex): + 1. log(I) = log(R) + log(L) + 2. log(R) = log(I) - log(Gaussian*I) + 3. exp(R), normalize to [0,255] + Accept: Unit test: image with shadow gradient → uniform illumination +``` + +#### 1.2.6 — Binarization: Sauvola Local Threshold + +``` +[1.2.6] Convert grayscale to clean black-and-white using adaptive threshold + Files: + apps/tools/backend/workers/src/scanner/binarize.rs + Functions: + compute_integral_image(img: &GrayImage) → Vec + compute_integral_image_sq(img: &GrayImage) -> Vec + local_stats(integral: &[u64], integral_sq: &[u64], + x: i32, y: i32, half_win: i32, w: i32, h: i32) -> (f64, f64) + sauvola_threshold(img: &GrayImage, window_size: u32, k: f64) -> GrayImage + otsu_threshold(img: &GrayImage) -> GrayImage // fallback + Algorithm: + Sauvola: T = m * (1 + k * (s/R - 1)) + - m = local mean (from integral image) + - s = local std dev (from integral image squared) + - k = 0.2 (tunable) + - R = 128 (max std dev for 8-bit) + - window_size = max(width, height) / 30, clamped to [15, 100] + Accept: Unit test: 5 test images → binary output, text readable, background clean white +``` + +#### 1.2.7 — Deskew: Hough Transform Line Detection + +``` +[1.2.7] Detect and correct small rotation (<5°) of text lines + Files: + apps/tools/backend/workers/src/scanner/deskew.rs + Functions: + probabilistic_hough_lines(img: &GrayImage, threshold: u32, + min_line_length: f64, max_gap: f64) -> Vec + median_angle(lines: &[Line]) -> f64 + rotate_image(img: &GrayImage, angle_degrees: f64) -> GrayImage + deskew(img: &GrayImage) -> GrayImage + Algorithm: + 1. Probabilistic Hough line transform + 2. Filter: keep lines with angle between -45° and +45° (skip vertical) + 3. Compute median angle + 4. If |angle| > 0.5° → rotate with Lanczos3, crop to fit + Accept: Unit test: rotated text image 3° → deskewed to <0.5° residual rotation +``` + +#### 1.2.8 — Image Enhancement: Sharpening + Contrast + +``` +[1.2.8] Apply final sharpening and contrast optimization + Files: + apps/tools/backend/workers/src/scanner/enhance.rs + Functions: + unsharp_mask(img: &GrayImage, sigma: f64, amount: f64) -> GrayImage + adjust_contrast(img: &GrayImage, factor: f64) -> GrayImage + remove_noise(img: &GrayImage, threshold: u8) -> GrayImage + enhance_final(img: &GrayImage) -> GrayImage + Algorithm (Unsharp mask): + blurred = gaussian_blur(img, sigma=1.0) + mask = img - blurred + result = img + amount * mask // amount = 1.0 (default) + Accept: Unit test: blurry text → sharpened text, verify no ringing artifacts +``` + +#### 1.2.9 — Pipeline Assembly + +``` +[1.2.9] Connect all pipeline stages with progress reporting + Files: + apps/tools/backend/workers/src/scanner/pipeline.rs + apps/tools/backend/workers/src/scanner/mod.rs (update) + Functions: + ScanPipeline::process(input_path, options, progress: ProgressReporter) → Result + Stages with progress: + 0% → load + preprocess + 15% → edge detection + 25% → corner detection + 35% → perspective warp + 50% → shadow removal + 65% → binarization + 75% → deskew + 85% → final enhance + 100% → complete + ScanResult: + { output_image_path, page_count: 1, image_dimensions, processing_time_ms } + Accept: Full pipeline test with 10 diverse test images → consistent quality output +``` --- -### Milestone 1.3 — Next.js Frontend +### Milestone 1.3 — Next.js Frontend Foundation -**Goal**: User bisa upload foto, liat progress, download hasil. +**Goal**: User bisa upload foto, lihat progress, download hasil. -**Tasks**: +#### 1.3.1 — Init Next.js App -| # | Task | Detail | -|---|------|--------| -| 1.3.1 | Init Next.js app | `bun create next-app` dengan Tailwind v4 + shadcn/ui | -| 1.3.2 | Landing page | Cards: Scan, Image, PDF — link ke masing-masing tool | -| 1.3.3 | Upload zone component | Drag & drop + file picker, validasi tipe/ukuran | -| 1.3.4 | Scanner page | `/scan` — upload area, tool options, progress bar | -| 1.3.5 | API upload route | Next.js API route → proxy ke Rust Gateway | -| 1.3.6 | Progress bar component | Animated bar + stage label dari WebSocket | -| 1.3.7 | Preview component | Before/after comparison slider | -| 1.3.8 | Result page | Preview + download button + file info | -| 1.3.9 | Error handling | Upload error, processing error, timeout | +``` +[1.3.1] Create Next.js 16 app with Tailwind v4 + shadcn/ui + TypeScript strict + Files: + apps/tools/frontend/package.json + apps/tools/frontend/next.config.ts + apps/tools/frontend/tsconfig.json + apps/tools/frontend/postcss.config.mjs + apps/tools/frontend/components.json + apps/tools/frontend/biome.json + apps/tools/frontend/src/app/globals.css + Setup: + bun create next-app@latest --typescript --tailwind --eslint + bun add @shadcn/react lucide-react class-variance-authority clsx tailwind-merge framer-motion + npx shadcn@latest init + Add custom CSS variables + Twilight Terminal theme + Accept: bun dev → localhost:3002, halaman kosong dengan Tailwind + shadcn working +``` -**Acceptance**: User upload foto → liat progress bar → download PDF. +#### 1.3.2 — Root Layout + Theme Provider -**Effort**: ~3-4 hari +``` +[1.3.2] Layout dengan header, footer, theme provider, fonts + Files: + apps/tools/frontend/src/app/layout.tsx + apps/tools/frontend/src/app/providers.tsx + apps/tools/frontend/src/components/tools/header.tsx + apps/tools/frontend/src/components/tools/footer.tsx + apps/tools/frontend/src/lib/utils.ts + Features: + - Root layout with metadata (title: "Tools — Asep Haryana") + - ThemeProvider (next-themes) wrapping children + - Geist sans font (same as hub portfolio) + - Header: logo "Tools", navigation links, theme toggle, GitHub link + - Footer: copyright, powered by Rust + Next.js badge + - cn() utility from tailwind-merge + Accept: All pages render with header + footer, theme toggle works +``` + +#### 1.3.3 — Landing Page (/) with Tool Cards + +``` +[1.3.3] Card grid showing all available tools with icons + Files: + apps/tools/frontend/src/app/page.tsx + apps/tools/frontend/src/components/tools/tool-card.tsx + apps/tools/frontend/src/components/tools/tool-grid.tsx + Data: + tools = [ + { id: "scan", title: "Document Scanner", desc: "...", icon: ScanIcon, href: "/scan", phase: 1 }, + { id: "image-compress", title: "Compress Image", desc: "...", icon: ... }, + ... + ] + Features: + - Grid responsive: 1 col mobile, 2 col tablet, 3 col desktop + - Each card: icon, title, description, link + - Phase badges: "Available", "Coming Soon" + - Framer Motion stagger animation on mount + Accept: / renders grid of tool cards, each card is clickable link +``` + +#### 1.3.4 — Upload Zone Component + +``` +[1.3.4] Drag & drop upload zone with file validation + Files: + apps/tools/frontend/src/components/tools/upload-zone.tsx + apps/tools/frontend/src/hooks/use-upload.ts + Features: + - Drag & drop area with dashed border + - Click to open file picker + - Accept attribute berdasarkan tool (image/*, application/pdf, video/*, audio/*) + - Validate: file type, max size (50MB), max count (50 for batch) + - Show file name, size, type after selection + - Error state: invalid type, too large, too many + - Drag over highlight animation + - Loading spinner during upload + - Upload progress percentage (from XHR or fetch) + - Cancel upload button + Accept: Drag image file → uploads to server → returns job_id +``` + +#### 1.3.5 — Progress Bar Component + +``` +[1.3.5] Animated progress bar with stage label from WebSocket + Files: + apps/tools/frontend/src/components/tools/progress-bar.tsx + apps/tools/frontend/src/hooks/use-job-status.ts + WebSocket hook (useJobStatus): + - Connect to /api/job/{id}/ws + - Auto-reconnect on disconnect (3 retries) + - Parse JobProgress messages + - Update state: status, progress, stage, message + - Cleanup on unmount + ProgressBar: + - Animated bar (Framer Motion width animation) + - Stage label: "Detecting edges...", "Correcting perspective...", etc. + - Percentage number + - Status badge: Processing (amber pulse), Completed (green), Failed (red) + - Error state with retry button + Accept: Upload → progress bar animates from 0-100% with stage labels +``` + +#### 1.3.6 — Preview Before/After Component + +``` +[1.3.6] Image preview with before/after comparison slider + Files: + apps/tools/frontend/src/components/tools/preview-before-after.tsx + Features: + - Two image layers: original (left) vs processed (right) + - Draggable slider divider + - Click on left = show original, click right = show processed + - Zoom: scroll to zoom, drag to pan + - File size comparison badge: "2.4 MB → 340 KB" + - Responsive: fill container width + Accept: Component renders with two image URLs, slider interaction works +``` + +#### 1.3.7 — Result Preview Component + +``` +[1.3.7] Result display: preview, download, info + Files: + apps/tools/frontend/src/components/tools/result-preview.tsx + Features: + - Show processed file preview (image or icon for PDF/video/audio) + - File info: name, size, dimensions, pages (for PDF) + - Download button with file type icon + - Download as ZIP for batch results + - Copy share link button (if applicable) + - "Process another" button → reset to upload state + - Auto-download option checkbox + Accept: Job completes → result card shows with download button → click downloads file +``` + +#### 1.3.8 — Scanner Page (/scan) + +``` +[1.3.8] Full scanner page: upload → progress → result + Files: + apps/tools/frontend/src/app/scan/page.tsx + apps/tools/frontend/src/app/scan/result/[id]/page.tsx + Page states: + 1. UPLOAD: UploadZone + options (OCR toggle, enhance toggle, DPI selector) + 2. PROCESSING: ProgressBar + stage label + cancel button + 3. RESULT: PreviewBeforeAfter + ResultPreview + "Process Another" + 4. ERROR: Error message with retry + feedback button + Options panel: + - Enable OCR (toggle, default: on) + - Auto-enhance (toggle, default: on) + - Output format (PDF, JPEG, PNG — default: PDF) + - DPI (150, 200, 300, 400 — default: 300) + Flow: + Upload → POST /api/upload → get job_id → connect WS → show progress → + complete → show result with preview + download + Accept: Full user flow: upload → progress → download PDF +``` + +#### 1.3.9 — API Route: Upload Proxy + +``` +[1.3.9] Next.js API route that proxies upload to Rust backend + Files: + apps/tools/frontend/src/app/api/upload/route.ts + Logic: + - Accept multipart/form-data from browser + - Forward to http://tools:3001/api/upload (Rust gateway) + - On 202: return { job_id, ws_url } to client + - On 4xx/5xx: return error to client + - Handle timeout, connection refused gracefully + Accept: POST via browser → proxied to Rust → returns job_id +``` + +#### 1.3.10 — API Route: Job Status & Download Proxy + +``` +[1.3.10] Proxy job status + WebSocket + download to Rust backend + Files: + apps/tools/frontend/src/app/api/job/[id]/route.ts + apps/tools/frontend/src/app/api/job/[id]/ws/route.ts + apps/tools/frontend/src/app/api/download/[id]/route.ts + Features: + GET /api/job/{id} → proxy to Rust + WS /api/job/{id}/ws → proxy WebSocket (Next.js can't do WS in app router, + so use upgrade header or direct client WS to Rust port) + GET /api/download/{id} → stream from Rust + Note: WebSocket langsung dari client ke Rust gateway port (3001), + bukan via Next.js. CORS sudah dihandle di Rust. + Accept: WS connection works: client → Rust gateway → NATS progress → browser +``` --- -### Milestone 1.4 — OCR + Searchable PDF +### Milestone 1.4 — OCR + Searchable PDF + Infrastructure -**Goal**: Output berupa PDF dengan hidden text layer — teks bisa di-copy, file bisa di-search. +**Goal**: Output searchable PDF, realtime WebSocket progress, auto-cleanup, rate limiting. -**Tasks**: +#### 1.4.1 — Tesseract OCR Integration -| # | Task | Detail | -|---|------|--------| -| 1.4.1 | Install Tesseract data | Tambah `tessdata` di Docker image | -| 1.4.2 | OCR integration | `leptess` binding, set language, get text + word boxes | -| 1.4.3 | PDF generation | `lopdf` — page with image + invisible text layer | -| 1.4.4 | WebSocket progress | NATS consumer di Gateway → broadcast ke WS client | -| 1.4.5 | Auto-cleanup scheduler | NATS cron tiap 10 menit, hapus file expired >1 jam | -| 1.4.6 | Rate limiting | Redis sliding window per IP, per tool | +``` +[1.4.1] OCR text extraction from processed image + Files: + apps/tools/backend/workers/src/scanner/ocr.rs + Functions: + init_tesseract(lang: &str) -> Result + ocr_text(tess: &mut LepTess, img: &GrayImage) -> Result + ocr_words(tess: &mut LepTess, img: &GrayImage) -> Result> + ocr_with_language(img: &GrayImage, lang: &str) -> Result + OcrWord: { text: String, bbox: {x,y,w,h}, confidence: i32 } + OcrResult: { full_text: String, words: Vec, confidence: f32 } + Languages: "eng+ind" (English + Indonesian) + TESSDATA_PREFIX: /usr/share/tesseract-ocr/5/tessdata + PSM mode: 3 (automatic), fallback 6 (single text block) + Accept: Unit test: known text image → OCR returns text with >80% confidence +``` -**Acceptance**: Download PDF → buka di browser → teks bisa di-select + di-search. +#### 1.4.2 — Searchable PDF Generation -**Effort**: ~3-4 hari +``` +[1.4.2] Generate PDF with visible image + invisible text layer + Files: + apps/tools/backend/workers/src/scanner/pdf.rs + Functions: + compress_image_jpeg(img: &GrayImage, quality: u8) -> Result> + generate_pdf_page(image_data: &[u8], words: &[OcrWord], + page_width_pt: f64, page_height_pt: f64) -> Result + generate_searchable_pdf(image_data: &[u8], ocr_text: &str, + words: &[OcrWord]) -> Result> + PDF structure: + - Page with MediaBox A4 (595.28 x 841.89) or fit to image aspect ratio + - Image XObject (JPEG DCTDecode, 300 DPI equivalent) + - Content stream: + 1. Place image at full page: q {w} 0 0 {h} 0 0 cm /Im0 Do Q + 2. Invisible text: 3 Tr (rendering mode 3 = neither fill nor stroke) + 3. Each word positioned at its bbox, converted pixels → points + - Metadata: Producer, CreationDate + Accept: Generated PDF → open in browser → text is selectable + searchable +``` + +#### 1.4.3 — Scanner Pipeline: OCR + PDF Integration + +``` +[1.4.3] Connect OCR + PDF generation into the main pipeline + Files: + apps/tools/backend/workers/src/scanner/pipeline.rs (update) + Updated stages: + 75% → deskew + 82% → OCR + 90% → PDF generation + 100% → save + complete + Pipeline now returns ScanResult: + { output_path, page_count, file_size, ocr_text, processing_time_ms } + Accept: Full pipeline test: input image → output searchable PDF file +``` + +#### 1.4.4 — WebSocket Progress Forwarding (Gateway) + +``` +[1.4.4] Gateway subscribes to NATS progress and forwards via WebSocket + Files: + apps/tools/backend/gateway/src/routes/ws.rs + Functions: + ws_handler(ws: WebSocketUpgrade, job_id: Path, state: SharedState) + handle_ws(mut ws: WebSocket, job_id: String, nats: Connection, redis: ConnectionManager) + Logic: + 1. Accept WebSocket upgrade + 2. Subscribe to NATS: tools.*.progress.{job_id} + 3. Forward each message as JSON to WebSocket client + 4. On connection close: unsubscribe from NATS + 5. Keepalive ping every 30 seconds + 6. Send initial status from Redis on connect + Accept: Client connects via WS → receives progress messages in realtime +``` + +#### 1.4.5 — Auto-Cleanup Scheduler + +``` +[1.4.5] Scheduled cleanup of expired files and Redis keys + Files: + apps/tools/backend/workers/src/scheduler/mod.rs + apps/tools/backend/workers/src/scheduler/cleanup.rs + Logic: + 1. Subscribe to NATS cron: tools.scheduler.cleanup (every 10 min) + 2. Scan {STORAGE_PATH} for files older than JOB_TTL_SECONDS + 3. Delete expired files + 4. SCAN Redis for job:* keys with TTL expired, delete orphans + 5. Log: deleted N files, freed M bytes, deleted K orphan keys + 6. Prometheus: tools_cleanup_deleted_files counter + Periodic trigger: NATS cron via tools.scheduler.cleanup subject + Accept: Upload file → wait TTL → file deleted automatically +``` + +#### 1.4.6 — Rate Limiting + +``` +[1.4.6] Rate limiting per IP per tool via Redis sliding window + Files: + apps/tools/backend/gateway/src/redis/ratelimit.rs (update) + apps/tools/backend/gateway/src/routes/upload.rs (middleware) + Algorithm (Sliding Window): + Key: ratelimit:{ip}:{tool}:{minute_bucket} + - ZADD current timestamp + - ZREMRANGEBYSCORE older than 60s + - ZCOUNT → if > RATE_LIMIT_PER_MINUTE → reject + Response on reject: 429 Too Many Requests + { error: "rate_limit_exceeded", retry_after_seconds: 60 } + Middleware: Add to upload route as tower Layer + Accept: curl 31x in 60s → 429 on 31st request +``` + +#### 1.4.7 — Error Handling & Validation (Gateway Middleware) + +``` +[1.4.7] Global error handling and input validation middleware + Files: + apps/tools/backend/gateway/src/middleware/mod.rs + apps/tools/backend/gateway/src/middleware/error_handler.rs + apps/tools/backend/gateway/src/middleware/request_id.rs + Features: + - Request ID middleware (X-Request-Id header, uuid v4) + - JSON error response format: { error: string, code: string, request_id: string } + - 400: invalid input (missing file, missing tool, invalid options) + - 404: job not found / file expired + - 413: file too large + - 429: rate limited + - 500: internal error (logged, not exposed to client) + - Panic recovery layer + Accept: curl with missing fields → 400 JSON error. curl bad tool → 400. +``` --- -### Phase 1 Total: ~14-19 hari kerja +### Phase 1 Complete: Document Scanner MVP + +**Acceptance criteria**: +1. User buka tools.asepharyana.my.id → landing page with tool cards +2. Click "Document Scanner" → halaman scan +3. Upload foto dokumen HP (miring, bayangan) → preview upload +4. Progress bar animasi: edge → warp → enhance → binarize → OCR → PDF +5. Download searchable PDF → teks bisa di-copy +6. Rate limit: 30 uploads/min/IP +7. File auto-delete after 1 hour --- ## Phase 2: Scanner Complete + Image Tools -### Milestone 2.1 — Scanner Fallback + PWA +### Milestone 2.1 — Scanner Robustness (Manual Crop + Camera + Batch) -**Goal**: Scanner robust — kalau auto gagal, user bisa atur manual. Kamera langsung dari browser. - -**Tasks**: - -| # | Task | Detail | -|---|------|--------| -| 2.1.1 | Manual crop UI | Canvas: 4 draggable corners, background image | -| 2.1.2 | Fallback pipeline | Auto → gagal → manual → kirim corners ke worker | -| 2.1.3 | Camera capture | PWA: akses kamera via `getUserMedia`, capture frame | -| 2.1.4 | Auto-exposure helper | Tap to focus + exposure lock | -| 2.1.5 | Batch multi-page | Upload multiple photos → 1 PDF result | - -**Effort**: ~4-5 hari - ---- - -### Milestone 2.2 — Image Tools - -**Goal**: Compress, resize, convert image langsung di browser (WASM). - -**Tasks**: - -| # | Task | Detail | -|---|------|--------| -| 2.2.1 | WASM image crate | Compile `image` crate to WASM via `wasm-pack` | -| 2.2.2 | Compress UI | Slider kualitas %, preview perbandingan ukuran | -| 2.2.3 | Resize UI | Input dimensi, lock aspect ratio, preview | -| 2.2.4 | Convert UI | Pilih format output, preview | -| 2.2.5 | Client-side processing | Semua image tool jalan di browser — no upload needed | -| 2.2.6 | Fallback server-side | Kalau WASM gagal/browser tua → upload ke server worker | - -**WASM Strategy**: -```rust -// apps/tools/backend/wasm/src/lib.rs -use wasm_bindgen::prelude::*; -use image::{DynamicImage, ImageFormat}; -use std::io::Cursor; - -#[wasm_bindgen] -pub fn compress_jpeg(bytes: &[u8], quality: u8) -> Vec { - let img = image::load_from_memory(bytes).unwrap(); - let mut output = Cursor::new(Vec::new()); - img.write_to(&mut output, ImageFormat::Jpeg).unwrap(); - // quality compression via mozjpeg or custom - output.into_inner() -} -``` - -**Effort**: ~4-5 hari - ---- - -### Milestone 2.3 — Background Removal - -**Goal**: Hapus background foto pake AI model ONNX — jalan di Rust native. - -**Tasks**: - -| # | Task | Detail | -|---|------|--------| -| 2.3.1 | Download RMBG model | `rmbg-1.4.onnx` (atau model lebih kecil seperti `u2net`) | -| 2.3.2 | ONNX Runtime binding | `ort` crate — load model, run inference | -| 2.3.3 | Pre/post processing | Resize ke 1024x1024 → normalize → softmax → threshold | -| 2.3.4 | Mask application | Alpha channel: background transparent / warna solid | -| 2.3.5 | Image preview | Before/after dengan background removal | - -**Model Options**: -| Model | Size | Quality | Notes | -|-------|------|---------|-------| -| RMBG-1.4 | ~50MB | Excellent | BRIA, butuh license untuk commercial | -| U-2-Net | ~170MB | Good | Open source, lebih besar | -| MODNet | ~25MB | Good | Ringan, cepat | -| Dis_seg | ~8MB | Decent | Paling kecil, cocok untuk VPS | - -**Effort**: ~3-4 hari - ---- - -### Milestone 2.4 — Batch Processing - -**Goal**: Upload 10-20 foto sekaligus, diproses parallel, jadi 1 PDF. - -**Tasks**: - -| # | Task | Detail | -|---|------|--------| -| 2.4.1 | Batch upload UI | Drop zone accept multiple files, thumbnail list | -| 2.4.2 | Group job | 1 group job = N individual jobs, track per-item progress | -| 2.4.3 | Rayon parallel | Worker process multiple pages in parallel | -| 2.4.4 | PDF merger | `lopdf` merge multiple pages → 1 document | - -**Effort**: ~3-4 hari - ---- - -## Phase 3: PDF + Video/Audio Tools - -### Milestone 3.1 — PDF Tools - -| # | Task | Detail | -|---|------|--------| -| 3.1.1 | Merge PDF | Upload 2+ PDF, `lopdf` merge pages | -| 3.1.2 | Split PDF | Input pages "1-3,5,7-9", ekstrak + save jadi 1 file | -| 3.1.3 | Images to PDF | Upload images, sort order, jadi 1 PDF | -| 3.1.4 | PDF compress | Re-encode embedded images dengan kualitas lebih rendah | -| 3.1.5 | PDF to images | Tiap halaman → JPEG/PNG | - -**Effort**: ~4-5 hari - ---- - -### Milestone 3.2 — Video/Audio Tools - -| # | Task | Detail | -|---|------|--------| -| 3.2.1 | Video compress | FFmpeg binding (`ffmpeg-next`), turunin bitrate + resolusi | -| 3.2.2 | Extract audio | MP4 → MP3 via FFmpeg | -| 3.2.3 | Trim video | Start/end time → cut segment | -| 3.2.4 | GIF maker | Video segment → GIF, atur FPS + dimensi | -| 3.2.5 | Audio convert | Format conversion via FFmpeg | - -**Catatan**: Video processing heavy — butuh dedicated worker dengan resource lebih besar. Queue priority: video jobs ke stream terpisah dengan max 1 concurrent. - -**Effort**: ~5-7 hari - ---- - -### Milestone 3.3 — Final Polish - -| # | Task | Detail | -|---|------|--------| -| 3.3.1 | Theme integration | Twilight Terminal theme dari hub | -| 3.3.2 | Responsive design | Mobile-first, touch-friendly crop | -| 3.3.3 | Error monitoring | Error tracking, alert kalau pipeline gagal | -| 3.3.4 | Load testing | k6: simulasi concurrent users, measure P50/P95/P99 latency | -| 3.3.5 | Dashboard integration | Link dari hub dashboard → tools stats | - -**Effort**: ~3-4 hari - ---- - -## Timeline Summary +#### 2.1.1 — Manual Crop Canvas ``` -Minggu 1: Gateway + upload/download + edge detection + warp -Minggu 2: Enhance + binarization + deskew + Next.js frontend -Minggu 3: OCR + PDF + WebSocket + rate limit + cleanup - ─── MVP LAUNCH (Document Scanner ready) ─── -Minggu 4: Manual crop fallback + PWA camera + batch -Minggu 5: WASM image tools + compress/resize/convert -Minggu 6: Background removal (ONNX) + PDF tools - ─── V1 LAUNCH (Scanner + Image + PDF) ─── -Minggu 7: Video/audio tools + final polish -Minggu 8: Load testing + bug fixes + deployment +[2.1.1] Interactive canvas with 4 draggable corner handles + Files: + apps/tools/frontend/src/components/tools/crop-editor.tsx + apps/tools/frontend/src/hooks/use-crop-editor.ts + Features: + - Render uploaded image on Canvas + - 4 draggable corner handles (circular, Luma color) + - Connect corners with dashed lines + - Zoom: scroll to zoom, drag canvas to pan + - Grid overlay (rule of thirds) for alignment + - Touch support: pinch zoom, drag handles + - Double-click to auto-detect corners again + - Reset button + - Confirm button → sends corner coordinates + image to server + API: POST /api/scan/manual-crop + { job_id, corners: [{x,y},{x,y},{x,y},{x,y}] } + Accept: User adjust corners → confirm → server applies warp with given corners ``` -## Critical Path +#### 2.1.2 — Fallback Pipeline: Auto → Manual → Process ``` -Edge Detection ──▶ Corner Detection ──▶ Perspective Warp - │ │ - │ ┌────────┘ - │ ▼ - │ Shadow Removal ──▶ Binarization ──▶ Deskew - │ │ - │ ┌─────────────┘ - │ ▼ - │ OCR ──▶ PDF Gen ──▶ Output - │ - └───(Kalau gagal)─── Manual Crop ◀── Frontend Canvas +[2.1.2] When auto edge detection fails, fall back to manual crop + Files: + apps/tools/backend/gateway/src/routes/upload.rs (update) + apps/tools/frontend/src/app/scan/page.tsx (update) + Flow: + 1. Upload → queue job → worker coba auto-detect + 2. Auto gagal → worker set status = "needs_manual_crop" + → return corner confidence < 0.6 + 3. Gateway returns: { job_id, status: "needs_manual_crop", + manual_crop_url: "/scan/crop/{job_id}" } + 4. Frontend redirect to CropEditor + 5. User adjust corners → POST manual-crop → worker resume pipeline + Accept: Upload bad photo → auto redirect to manual crop → complete pipeline ``` -**Risks**: -1. Edge detection paling rentan gagal — pipeline harus graceful fallback ke manual crop -2. Homography SVD implementasi perlu numerik stabil — test dengan extreme perspective angles -3. OCR kualitas sangat tergantung pada binarization — Sauvola parameter perlu tuning -4. WASM image processing size besar (~2MB gzipped) — perlu code splitting + lazy load +#### 2.1.3 — PWA Camera Capture + +``` +[2.1.3] Direct camera capture from browser (PWA) + Files: + apps/tools/frontend/src/components/tools/camera-capture.tsx + apps/tools/frontend/src/hooks/use-camera.ts + Features: + - Access rear camera via getUserMedia + - Live preview in viewfinder + - Auto-focus on tap + - Capture button → freeze frame + - Aspect ratio guide overlay (A4: 1:1.414) + - Reject blurry photos (Laplacian variance check via Canvas API) + - Auto-capture when document detected (stabilize → snap) + - Switch front/rear camera + - Torch/flash toggle (if supported) + - Zoom slider (if supported) + Accept: Click "Use Camera" → camera opens → capture → photo uploaded +``` + +#### 2.1.4 — Batch Multi-Page + +``` +[2.1.4] Multiple pages → one PDF, parallel processing + Files: + apps/tools/frontend/src/components/tools/file-list.tsx + apps/tools/backend/workers/src/scanner/pipeline.rs (update) + Frontend: + - Upload multiple files (drag multiple or multi-select) + - Thumbnail list with drag-to-reorder + - Remove individual pages + - Add more pages button + - Upload all → single group job → N individual jobs + - Per-page progress status in thumbnail list + Backend: + - Group job: { group_id, job_ids: [...], total: N, completed: 0 } + - Process each page via Rayon parallel for + - Merge all PDF pages into single document via lopdf + - Report: per-page progress + overall progress % (completed/total) + Accept: Upload 5 photos → all processed in parallel → single 5-page PDF +``` + +#### 2.1.5 — Scan Options Panel + +``` +[2.1.5] Enhanced options panel for scanner + Files: + apps/tools/frontend/src/components/tools/scan-options.tsx + apps/tools/frontend/src/app/scan/page.tsx (update) + Options: + - Output: PDF, JPEG, PNG + - Quality: 1-100 (slider, default 90) + - OCR: on/off (default: on) + - Language: English, Indonesian, Both (default: Both) + - Auto-enhance: on/off (default: on) + - Color mode: Black & White, Grayscale, Color (default: B&W) + - Page size: A4, Letter, Auto-fit (default: A4) + - DPI: 150/200/300/400 (default: 300) + Accept: Change options → upload → output sesuai pilihan +``` + +--- + +### Milestone 2.2 — WASM Image Tools (Compress, Resize, Convert) + +#### 2.2.1 — WASM Rust Crate + +``` +[2.2.1] Compile image processing to WebAssembly via wasm-pack + Files: + apps/tools/backend/wasm/Cargo.toml + apps/tools/backend/wasm/src/lib.rs + apps/tools/backend/wasm/src/compress.rs + apps/tools/backend/wasm/src/resize.rs + apps/tools/backend/wasm/src/convert.rs + Functions: + #[wasm_bindgen] + fn compress_jpeg(bytes: &[u8], quality: u8) -> Result> + fn compress_png(bytes: &[u8], effort: u8) -> Result> + fn compress_webp(bytes: &[u8], quality: u8) -> Result> + fn resize_image(bytes: &[u8], width: u32, height: u32, fit: String) -> Result> + fn convert_format(bytes: &[u8], target_format: String) -> Result> + fn get_image_info(bytes: &[u8]) -> Result // { width, height, format, size } + Build: wasm-pack build --release --target web + Output: pkg/ directory (wasm binary + JS glue) + Accept: wasm-pack build succeeds. Node.js test: compress results smaller than input +``` + +#### 2.2.2 — WASM Integration in Frontend + +``` +[2.2.2] Load and invoke WASM module from Next.js + Files: + apps/tools/frontend/src/lib/wasm/loader.ts + apps/tools/frontend/src/lib/wasm/image-processor.ts + Functions: + async initWasm() → WASM module instance (lazy load on first use) + async compressInBrowser(file: File, quality: number) → Blob + async resizeInBrowser(file: File, width: number, height: number) → Blob + async convertInBrowser(file: File, format: string) → Blob + Strategy: + - Dynamic import: await import('./pkg/image_wasm.js') + - Lazy init: only load when user visits image tool page + - Code splitting: WASM chunk loaded separately (~1.5MB gzipped) + - Cache: once loaded, keep in memory + - Fallback: if WASM fails → upload to server worker + Accept: Browser loads WASM → image processing runs client-side, no upload +``` + +#### 2.2.3 — Image Compress Page (/image/compress) + +``` +[2.2.3] Full compress page with quality slider and live comparison + Files: + apps/tools/frontend/src/app/image/compress/page.tsx + apps/tools/frontend/src/components/tools/quality-slider.tsx + Features: + - UploadZone (accepts image/*) + - Quality slider: 1-100, live preview update + - File size display: original vs compressed (estimated) + - PreviewBeforeAfter (original vs compressed) + - Format selector: JPEG, PNG, WebP + - Download button + - Batch mode: compress all images in folder + WASM flow: upload → load in WASM → compress → preview → download (no server) + Server fallback: upload → Rust worker compress → download + Accept: Upload photo → adjust quality → live preview → download compressed version +``` + +#### 2.2.4 — Image Resize Page (/image/resize) + +``` +[2.2.4] Resize page with dimension input and aspect ratio lock + Files: + apps/tools/frontend/src/app/image/resize/page.tsx + apps/tools/frontend/src/components/tools/dimension-input.tsx + Features: + - UploadZone + - Input: width + height, auto-fill from original + - Aspect ratio lock toggle (🔗/🔓) + - Preset sizes: 800x600, 1024x768, 1920x1080, Instagram (1080x1080), Custom + - Fit modes: exact (stretch), contain (fit within, add bg), cover (crop to fill) + - Preview: resized dimensions overlay + - Download (or ZIP for batch) + - Batch: resize all images to same dimensions + Accept: Upload photo → set 800px width → lock aspect → download resized image +``` + +#### 2.2.5 — Image Convert Page (/image/convert) + +``` +[2.2.5] Format conversion page + Files: + apps/tools/frontend/src/app/image/convert/page.tsx + Features: + - UploadZone + - From: auto-detected from file + - To: JPEG, PNG, WebP, GIF, BMP, TIFF + - Quality slider (for lossy formats) + - PreviewBeforeAfter + - Download + - Batch: convert all files in folder + Accept: Upload HEIC → convert to JPEG → download +``` + +#### 2.2.6 — Server Fallback for WASM + +``` +[2.2.6] Server-side processing when WASM unavailable + Files: + apps/tools/backend/workers/src/image/mod.rs + apps/tools/backend/workers/src/image/compress.rs + apps/tools/backend/workers/src/image/resize.rs + apps/tools/backend/workers/src/image/convert.rs + Logic: + - Same pipeline as WASM but runs natively + - Use image crate with mozjpeg feature for optimal JPEG + - Resize with Lanczos3 filter + - Convert via image crate format support + Frontend detection: + - try/catch WASM init → if fails, enable "Server Process" button + - Upload → NATS → worker → result (same as scanner flow) + Accept: Disable WASM in browser → upload fallback works → same quality output +``` + +--- + +### Milestone 2.3 — Background Removal (ONNX) + +#### 2.3.1 — ONNX Model Download & Setup + +``` +[2.3.1] Download and package background removal model + Files: + Dockerfile (update — or runtime download script) + models/download-models.sh + Models: + - Primary: MODNet (~25MB, good quality, fast) + - Alternative: DIS_seg (~8MB, decent quality, very fast) + Setup: + - Download model file to models/ + - Verify SHA256 checksum + - Load model at worker startup + - One-time init, cache in memory + ONNX session options: + - InterOpNumThreads: 2 + - IntraOpNumThreads: 4 + - GraphOptimizationLevel: ORT_ENABLE_ALL + Accept: Worker starts → loads ONNX model → ready for inference +``` + +#### 2.3.2 — Pre/Post Processing + +``` +[2.3.2] Image preprocessing and postprocessing for ONNX model + Files: + apps/tools/backend/workers/src/image/remove_bg.rs + Functions: + preprocess_for_model(img: &DynamicImage, target_size: u32) -> Result> + - Resize to model input size (1024x1024 for MODNet) + - Normalize: /255.0, mean=[0.5,0.5,0.5], std=[0.5,0.5,0.5] + - Convert to CHW format (ort::Tensor) + postprocess_mask(output: &ort::Tensor, original_size: (u32,u32)) -> GrayImage + - Sigmoid → threshold 0.5 → resize to original dimensions + - Apply optional smoothing (Gaussian blur sigma=1) + apply_alpha(img: &DynamicImage, mask: &GrayImage, bg_color: Option<[u8;3]>) -> DynamicImage + - If bg_color = None → RGBA with transparent background + - If bg_color = Some([r,g,b]) → composite onto solid color + Accept: Unit test: portrait photo → mask correctly separates foreground +``` + +#### 2.3.3 — Remove Background Page (/image/remove-bg) + +``` +[2.3.3] Background removal page + Files: + apps/tools/frontend/src/app/image/remove-bg/page.tsx + Features: + - UploadZone + - Processing via server worker (too heavy for WASM) + - PreviewBeforeAfter: original → transparent bg + - Background selector: transparent, white, color picker + - Download as PNG (transparent) or JPEG (with bg color) + - Download HD (full resolution) or Web (compressed) + - Batch processing + - Progress bar during inference + Accept: Upload photo → background removed → download with transparent PNG +``` + +--- + +### Phase 2 Complete: Document Scanner Robust + Image Tools + +--- + +## Phase 3: PDF Tools + +### Milestone 3.1 — PDF Worker + +#### 3.1.1 — PDF Merge Worker + +``` +[3.1.1] Merge multiple PDF files into one + Files: + apps/tools/backend/workers/src/pdf/mod.rs + apps/tools/backend/workers/src/pdf/merge.rs + apps/tools/frontend/src/app/pdf/merge/page.tsx + Logic: + 1. Upload 2+ PDF files + 2. Load each via lopdf::Document::load + 3. Iterate pages from each document, append to output + 4. Save merged document + 5. Preserve: page size (use largest MediaBox per page), fonts (embedded) + Frontend: + - UploadZone (multiple, accept .pdf) + - Drag-to-reorder uploaded files + - Remove individual files + - Download merged PDF + Accept: Upload 3 PDFs → download 1 PDF with all pages in order +``` + +#### 3.1.2 — PDF Split Worker + +``` +[3.1.2] Extract specific pages from a PDF + Files: + apps/tools/backend/workers/src/pdf/split.rs + apps/tools/frontend/src/app/pdf/split/page.tsx + Functions: + parse_page_range(input: &str, total_pages: u32) -> Result> + split_pdf(input_path: &Path, pages: &[u32]) -> Result> + Page syntax: "1-3,5,7-9,12" → [1,2,3,5,7,8,9,12] + Frontend: + - UploadZone (single PDF) + - PDF preview with page thumbnails + - Click pages to select/deselect + - Or type page range input + - Download extracted pages as single PDF + Accept: Upload 20-page PDF → extract pages 3-7 → download 5-page PDF +``` + +#### 3.1.3 — Images to PDF Worker + +``` +[3.1.3] Convert multiple images into a single PDF + Files: + apps/tools/backend/workers/src/pdf/images_to_pdf.rs + apps/tools/frontend/src/app/pdf/images-to-pdf/page.tsx + Functions: + images_to_pdf(image_paths: &[PathBuf], options: PdfOptions) -> Result> + Logic: + 1. Load each image via image::open + 2. Convert to JPEG (for PDF embedding) + 3. Create PDF page per image + 4. Fit image to A4 / Letter / Original size + 5. Optional: margin, alignment + Frontend: + - UploadZone (multiple, accept image/*) + - Thumbnail grid with drag-to-reorder + - Page size: A4, Letter, Original + - Orientation: Auto, Portrait, Landscape + - Margin: 0-50mm + - Download PDF + Accept: Upload 5 photos → download 1 PDF, each photo = 1 page +``` + +#### 3.1.4 — PDF Compress Worker + +``` +[3.1.4] Reduce PDF file size by recompressing embedded images + Files: + apps/tools/backend/workers/src/pdf/compress.rs + apps/tools/frontend/src/app/pdf/compress/page.tsx + Algorithm: + 1. Open PDF with lopdf + 2. Find all image XObject streams + 3. For each: decode → re-encode JPEG with lower quality + 4. Replace stream in PDF + 5. Remove unused objects + 6. Linearize PDF for fast web viewing + Compression levels: + - Maximum (q=30): smallest size, visible quality loss + - Balanced (q=50): good balance (default) + - Minimal (q=80): slight size reduction, near-lossless + Frontend: + - UploadZone (single PDF) + - Compression level selector + - Preview: original size vs estimated compressed size + - Download compressed PDF + Accept: Upload 10MB PDF with images → compress → <3MB output +``` + +#### 3.1.5 — PDF to Images Worker + +``` +[3.1.5] Convert each PDF page to an image + Files: + apps/tools/backend/workers/src/pdf/to_images.rs + apps/tools/frontend/src/app/pdf/pdf-to-images/page.tsx + Functions: + pdf_page_to_image(input_path: &Path, page_num: u32, dpi: u32) -> Result> + pdf_to_images(input_path: &Path, dpi: u32, format: &str) -> Result>> + Logic: + 1. Render PDF page to image (via lopdf rasterize or poppler) + 2. Output format: JPEG, PNG, WebP + 3. DPI: 72, 150, 200, 300 (default: 200) + Frontend: + - UploadZone (single PDF) + - DPI selector + - Format selector + - Preview: page thumbnails + - Download as ZIP (all images) or per-page + Accept: Upload PDF → download ZIP of page images +``` + +--- + +## Phase 4: Video/Audio Tools + +### Milestone 4.1 — FFmpeg Worker + +#### 4.1.1 — FFmpeg Binding & Worker Setup + +``` +[4.1.1] Integrate FFmpeg via ffmpeg-next crate + Files: + apps/tools/backend/workers/Cargo.toml (add ffmpeg-next) + apps/tools/backend/workers/src/video/mod.rs + apps/tools/backend/workers/src/audio/mod.rs + Dockerfile (update — install ffmpeg package) + Setup: + - apt-get install ffmpeg + - ffmpeg-next version 7.x + - Initialize: ffmpeg::init() at worker startup + - Test: version check + Accept: Worker starts → ffmpeg initialized → transcoding functions available +``` + +#### 4.1.2 — Video Compress + +``` +[4.1.2] Reduce video file size by lowering bitrate and/or resolution + Files: + apps/tools/backend/workers/src/video/compress.rs + apps/tools/frontend/src/app/video/compress/page.tsx + Options: + - Target size (approximate): 10MB, 25MB, 50MB, 100MB, Custom + - Resolution: 480p, 720p, 1080p, Original + - Quality: Low, Medium, High (CRF: 28, 23, 18) + - Codec: H.264, H.265/HEVC, VP9 + FFmpeg command (via ffmpeg-next API): + - Transcode video stream: libx264, crf, preset medium + - Scale if needed: scale=iw/2:ih/2 + - Copy audio stream (or compress with aac) + Frontend: + - UploadZone (accept video/*, max 500MB) + - Options: target size, resolution, quality + - Progress: from FFmpeg stderr parsed progress + - Preview: thumbnail + duration, original vs estimated size + - Download compressed video + Note: Video processing is HEAVY → max 1 concurrent video job, queue via NATS + Accept: Upload 500MB 1080p video → compress → <100MB 720p output +``` + +#### 4.1.3 — Audio Extract + +``` +[4.1.3] Extract audio track from video file + Files: + apps/tools/backend/workers/src/video/audio_extract.rs + apps/tools/frontend/src/app/video/audio-extract/page.tsx + Options: + - Format: MP3, AAC, WAV, FLAC, OGG + - Quality: 128k, 192k, 320k (for lossy) / Lossless (for FLAC/WAV) + FFmpeg: + - Stream copy if format compatible, else transcode + - Extract best audio stream + Frontend: + - UploadZone (accept video/*) + - Format + quality selector + - Progress + - Download extracted audio + Accept: Upload MP4 → download MP3 with correct audio +``` + +#### 4.1.4 — Video Trim + +``` +[4.1.4] Cut a segment from a video + Files: + apps/tools/backend/workers/src/video/trim.rs + apps/tools/frontend/src/app/video/trim/page.tsx + Options: + - Start time (HH:MM:SS or seconds) + - End time / Duration + FFmpeg: + -ffmpeg -i input -ss start -to end -c copy output (fast seek) + Or re-encode for precise seeking: -ss start -i input -t duration -c libx264 + Frontend: + - UploadZone (accept video/*) + - Preview player with timeline + - Set start/end via sliders or input + - Preview trim result + - Download trimmed video + Accept: Upload 10min video → trim 2:30-5:00 → download 2.5min video +``` + +#### 4.1.5 — GIF Maker + +``` +[4.1.5] Convert video segment to animated GIF + Files: + apps/tools/backend/workers/src/video/gif.rs + apps/tools/frontend/src/app/video/gif-maker/page.tsx + Options: + - Start time, Duration (max 10s for GIF) + - FPS: 10, 15, 20, 24, 30 + - Width: 320, 480, 640, 800 + - Dither: on/off + - Colors: 64, 128, 256 + FFmpeg: + ffmpeg -i input -ss start -t duration -vf "fps=15,scale=480:-1:flags=lanczos,palettegen" palette.png + ffmpeg -i input -i palette.png -ss start -t duration -lavfi "fps=15,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif + Frontend: + - UploadZone (accept video/*) + - Start time + duration sliders + - FPS + width + dither options + - Preview: animated GIF in browser + - Download GIF + Accept: Upload 30s video → trim 2s → download animated GIF +``` + +#### 4.1.6 — Audio Convert + +``` +[4.1.6] Convert audio between formats + Files: + apps/tools/backend/workers/src/audio/convert.rs + apps/tools/frontend/src/app/audio/convert/page.tsx + Formats: + Input: MP3, WAV, FLAC, AAC, OGG, M4A, WMA + Output: MP3, WAV, FLAC, AAC, OGG + Options: + - Bitrate: 128k, 192k, 256k, 320k + - Sample rate: 44100, 48000, 96000 + - Channels: Mono, Stereo (downmix if source is 5.1) + Frontend: + - UploadZone (accept audio/*) + - From (auto-detected) → To selector + - Bitrate + sample rate options + - Download converted audio + Accept: Upload FLAC → convert to 320kbps MP3 → download +``` + +--- + +## Phase 5: Deployment & Polish + +### Milestone 5.1 — Infrastructure + +#### 5.1.1 — Docker Multi-Stage Build + +``` +[5.1.1] Production Dockerfile + Files: + infra/docker/tools.Dockerfile + Stages: + 1. chef (cargo-chef install) + 2. planner (recipe.json) + 3. builder (cargo build —release) + 4. wasm-builder (wasm-pack build) + 5. frontend-builder (bun build) + 6. runtime: debian:bookworm-slim + tesseract + ffmpeg + ONNX model + Runtime dependencies: + - tesseract-ocr + tessdata (eng, ind) + - ffmpeg + - ca-certificates + - libfontconfig1 (for lopdf) + Dockerignore: + - node_modules, target, .next, .git + Build: docker build -f infra/docker/tools.Dockerfile -t tools:latest . + Accept: docker build succeeds, image size <400MB +``` + +#### 5.1.2 — Docker Compose File + +``` +[5.1.2] Compose file for production deployment + Files: + infra/compose/tools.yml + Service definition: + container_name: tools + image: ghcr.io/asepharyana/asepharyana-hub/tools:sha-xxxxx + restart: always + networks: app-shared-net (alias: tools) + env_file: ../../.env + volumes: tools_data:/data/tools + ports: 3001:3001 + depends_on: [redis, nats] + labels: prometheus.io/scrape=true, prometheus.io/port=3001 + Volume: tools_data (docker volume) + Accept: docker compose up → tools container running, connected to redis + nats +``` + +#### 5.1.3 — Traefik Routing + +``` +[5.1.3] Add Traefik router and service for tools + Files: + infra/traefik/dynamic/apps.yaml (update) + Router: + tools: + rule: Host(`tools.asepharyana.my.id`) || Host(`tools.asepharyana.web.id`) + entryPoints: websecure + tls: {} + middlewares: common-chain@file + service: tools-service + Service: + tools-service: + loadBalancer: + servers: + - url: http://tools:3001 + Accept: tools.asepharyana.my.id → loads tools frontend +``` + +#### 5.1.4 — CI/CD Workflow Integration + +``` +[5.1.4] Add tools service to existing build + deploy workflows + Files: + .github/workflows/docker-build-push.yml (update) + .github/workflows/deploy-docker.yml (check — auto-detects compose changes) + Changes: + - Detect changed service (apps/tools/**) + - Build matrix: add tools service + - Dockerfile: tools.Dockerfile + - Path: apps/tools + - Compose file: tools.yml + - Update manifest: sed image tag in compose + Accept: Push to main with apps/tools changes → CI builds + deploys tools +``` + +#### 5.1.5 — Environment Variables Setup + +``` +[5.1.5] Add tools env vars to .env.example + Files: + .env.example (update) + Vars: + # Tools + TOOLS_GATEWAY_PORT=3001 + TOOLS_WORKER_CONCURRENCY=4 + TOOLS_STORAGE_PATH=/data/tools + TOOLS_JOB_TTL_SECONDS=3600 + TOOLS_RATE_LIMIT_PER_MINUTE=30 + TOOLS_MAX_FILE_SIZE_MB=50 + TOOLS_OCR_LANG=eng+ind + Accept: .env.example updated with tools section +``` + +--- + +### Milestone 5.2 — Frontend Polish + +#### 5.2.1 — Theme Integration + +``` +[5.2.1] Apply Twilight Terminal theme consistent with portfolio hub + Files: + apps/tools/frontend/src/app/globals.css (update) + Theme vars (from hub): + --background / --foreground + --primary / --primary-foreground + --card / --card-foreground + --muted / --muted-foreground + Glass effect: .glass { backdrop-filter: blur } + Gradient text: .gradient-text + Terminal cursor blink animation + Same dark/light mode switch mechanism + Accept: tools subdomain → visual style consistent with hub portfolio +``` + +#### 5.2.2 — Responsive Mobile Design + +``` +[5.2.2] All pages responsive for mobile devices + Files: All page/component files (review) + Requirements: + - UploadZone: full-width on mobile, tap-friendly + - Tool cards: single column on mobile + - ProgressBar: always visible, top-fixed on scroll + - CropEditor: touch-drag handles, pinch-zoom + - CameraCapture: fullscreen viewfinder + - FileList: compact thumbnail list on mobile + - Buttons: min 44px touch target + - Bottom sheet instead of modal for options + - Safe area insets for notch devices + Accept: Lighthouse mobile audit >80 for all pages +``` + +#### 5.2.3 — Loading States & Skeleton + +``` +[5.2.3] Skeleton loading states for all pages + Files: + apps/tools/frontend/src/components/tools/skeleton.tsx + apps/tools/frontend/src/app/scan/page.tsx (update) + apps/tools/frontend/src/app/image/compress/page.tsx (update) + (all other tool pages) + Components: + SkeletonCard (pulse animation) + SkeletonUploadZone + SkeletonProgressBar + SkeletonPreview + Accept: All pages show skeleton while loading data/WASM +``` + +#### 5.2.4 — Error Boundaries + +``` +[5.2.4] React error boundaries per page + global + Files: + apps/tools/frontend/src/components/tools/error-boundary.tsx + apps/tools/frontend/src/app/layout.tsx (wrap with ErrorBoundary) + Each tool page: wrap with ErrorBoundary + Behavior: + - Catch React render errors + - Show friendly error message with tool name + - "Try Again" button + - "Report Issue" link (GitHub) + - Log error details to console (future: telemetry) + Accept: Force render error → error boundary shows, app doesn't crash +``` + +#### 5.2.5 — PWA Manifest + +``` +[5.2.5] Progressive Web App configuration + Files: + apps/tools/frontend/public/manifest.json + apps/tools/frontend/src/app/layout.tsx (add manifest link + meta tags) + apps/tools/frontend/public/icons/ (app icons: 192x192, 512x512) + Manifest: + name: "Tools — Asep Haryana" + short_name: "Tools" + description: "Document Scanner, Image & PDF Tools" + start_url: / + display: standalone + background_color: #0a0a1a (dark theme) + theme_color: #0a0a1a + icons: 192x192, 512x512 + Accept: Lighthouse PWA audit >80 +``` + +--- + +### Milestone 5.3 — Monitoring & Observability + +#### 5.3.1 — Prometheus Alerts + +``` +[5.3.1] Alert rules for tools service + Files: + infra/otel/prometheus.yml (update — or separate alert file) + Rules: + - High error rate: rate(tools_jobs_total{status="failed"}[5m]) > 0.1 + - Queue buildup: tools_queue_depth > 50 + - Slow processing: tools_processing_duration_ms{quantile="0.95"} > 10000 + - Low disk space: (disk_free_bytes / disk_total_bytes) < 0.1 (if node_exporter) + Accept: Rules loaded in Prometheus, alert firing correctly +``` + +#### 5.3.2 — Dashboard Integration + +``` +[5.3.2] Add tools metrics to hub dashboard + Files: + apps/hub/src/app/dashboard/page.tsx (update) + apps/hub/src/app/api/dashboard/route.ts (update) + Add to dashboard: + - Card: "Tools" service status (running/degraded/down) + - Quick stats: Total jobs today, Active jobs, Storage used + - Link to tools.asepharyana.my.id + Accept: Dashboard shows tools service status and stats +``` + +#### 5.3.3 — Structured Logging + +``` +[5.3.3] Structured JSON logging for production + Files: + apps/tools/backend/common/src/logging.rs (or in each crate) + apps/tools/backend/gateway/src/main.rs (logging init) + apps/tools/backend/workers/src/main.rs (logging init) + Config: + - Default: human-readable (development) + - JSON mode: RUST_LOG_FORMAT=json (production) + Fields per log: + - timestamp (ISO 8601) + - level (INFO, WARN, ERROR) + - service (gateway / workers / pipeline) + - request_id (if within request context) + - job_id (if within job context) + - message + - duration_ms (for completed processing) + Accept: RUST_LOG_FORMAT=json → JSON-structured log output +``` + +--- + +## Effort Summary + +| Phase | Milestone | Tasks | Estimated Hours | Total Days | +|-------|-----------|-------|-----------------|------------| +| 1 | 1.1 Rust Backend Skeleton | 15 | ~45 | 6 | +| 1 | 1.2 Scanner Pipeline Core | 9 | ~50 | 7 | +| 1 | 1.3 Next.js Frontend | 10 | ~30 | 4 | +| 1 | 1.4 OCR + PDF + Infra | 7 | ~25 | 4 | +| **Phase 1 Total** | **41** | **~150** | **~21** | +| 2 | 2.1 Scanner Robustness | 5 | ~25 | 4 | +| 2 | 2.2 WASM Image Tools | 6 | ~30 | 4 | +| 2 | 2.3 Background Removal | 3 | ~15 | 2 | +| **Phase 2 Total** | **14** | **~70** | **~10** | +| 3 | 3.1 PDF Tools | 5 | ~25 | 4 | +| **Phase 3 Total** | **5** | **~25** | **~4** | +| 4 | 4.1 FFmpeg Worker | 6 | ~35 | 5 | +| **Phase 4 Total** | **6** | **~35** | **~5** | +| 5 | 5.1 Infrastructure | 5 | ~15 | 2 | +| 5 | 5.2 Frontend Polish | 5 | ~20 | 3 | +| 5 | 5.3 Monitoring | 3 | ~10 | 2 | +| **Phase 5 Total** | **13** | **~45** | **~7** | +| **Grand Total** | **79 tasks** | **~325 hours** | **~47 days** | + +> **MVP** (Phase 1 only): 41 tasks, ~21 days +> **Full release** (Phase 1-5): 79 tasks, ~47 days + +--- + +## Critical Path (Phase 1) + +``` +Day 1-2: 1.1.1 → 1.1.2 → 1.1.3 → 1.1.4 → 1.1.5 (workspace + common) +Day 3-5: 1.1.6 → 1.1.7 → 1.1.8 → 1.1.10 → 1.1.14 (gateway routes + redis) +Day 5-6: 1.1.9 → 1.1.11 → 1.1.12 → 1.1.13 (gateway + workers connect) +Day 7-10: 1.2.1 → 1.2.2 → 1.2.3 → 1.2.4 (edge → warp) +Day 10-12: 1.2.5 → 1.2.6 → 1.2.7 → 1.2.8 (shadow → binarize → enhance) +Day 12: 1.2.9 (pipeline assembly) +Day 13-16: 1.3.1 → 1.3.2 → 1.3.3 → 1.3.4 (frontend pages) +Day 14-17: 1.3.5 → 1.3.6 → 1.3.7 → 1.3.8 (components) +Day 15-17: 1.3.9 → 1.3.10 (API proxy routes) +Day 18-19: 1.4.1 → 1.4.2 → 1.4.3 (OCR + PDF) +Day 19: 1.4.4 (WebSocket) +Day 20: 1.4.5 → 1.4.6 → 1.4.7 (cleanup + rate limit + errors) +``` + +> **MVP launch**: Day ~21 — Document Scanner live di tools.asepharyana.my.id \ No newline at end of file