asepharyana-hub/docs/plan/tools/implementation.md

Implementation Plan

Phase Breakdown

Phase 1 — Foundation + Document Scanner MVP
├── Milestone 1.1: Rust backend skeleton (Gateway + Worker + NATS + Redis)
├── Milestone 1.2: Scanner pipeline core (edge → warp → enhance → B&W)
├── Milestone 1.3: Next.js frontend + upload + download
└── Milestone 1.4: OCR + searchable PDF + WebSocket progress

Phase 2 — Scanner Complete + Image Tools
├── Milestone 2.1: Scanner fallback manual crop + PWA camera
├── Milestone 2.2: Image compress, resize, convert (WASM client-side)
├── Milestone 2.3: Background removal (ONNX)
└── Milestone 2.4: Batch processing

Phase 3 — PDF Tools + Video/Audio
├── Milestone 3.1: PDF merge, split, images-to-pdf
├── Milestone 3.2: Video compress, extract audio
└── Milestone 3.3: Final polish + load testing

---

Phase 1: Foundation + Document Scanner MVP

Milestone 1.1 — Rust Backend Skeleton

Goal: Gateway + Worker bisa connected ke NATS + Redis, upload flow work.

Tasks:

#	Task	Files	Detail
1.1.1	Init Rust workspace	apps/tools/backend/Cargo.toml	Workspace dengan 3 crate: gateway, workers, common
1.1.2	Common types	common/src/types.rs	JobStatus, Job, Tool, ScanOptions, UploadResponse
1.1.3	Common error	common/src/error.rs	PipelineError, UploadError, NatsError
1.1.4	NATS subjects	common/src/nats.rs	Constants untuk semua subject/stream
1.1.5	Gateway: config	gateway/src/config.rs	Env-based config (Redis URL, NATS URL, storage path)
1.1.6	Gateway: upload route	gateway/src/routes/upload.rs	Multipart upload, validasi, save ke temp, publish NATS
1.1.7	Gateway: job status	gateway/src/routes/job.rs	GET job status dari Redis
1.1.8	Gateway: download	gateway/src/routes/download.rs	Stream file dari storage
1.1.9	Gateway: NATS publish	gateway/src/nats/publisher.rs	Publish job + progress
1.1.10	Gateway: Redis job	gateway/src/redis/job.rs	CRUD job status di Redis
1.1.11	Gateway: main	gateway/src/main.rs	Axum app bootstrap + routes
1.1.12	Workers: main loop	workers/src/main.rs	NATS consumer, dispatch ke tool handler
1.1.13	Workers: NATS consumer	workers/src/nats/consumer.rs	Subscribe jobs queue, ack/nack
1.1.14	Worker: scanner stub	workers/src/scanner/mod.rs	Cuma menerima job, update progress, complete

Acceptance: curl -X POST -F "file=@test.jpg" -F "tool=scan" http://localhost:3001/api/upload → return job_id, setelah beberapa detik GET /api/job/:id return completed.

Effort: ~3-4 hari

---

Milestone 1.2 — Scanner Pipeline Core

Goal: Image processing pipeline bisa ngubah foto miring jadi lurus + bersih + hitam-putih. Belum termasuk OCR dan PDF.

Tasks:

#	Task	Detail	Referensi
1.2.1	Edge detection	Canny + morphological close + adaptive fallback	pipeline.md Stage 2-3
1.2.2	Corner detection	Contour detection, largest rectangle filter, polygon approximation	pipeline.md Stage 3
1.2.3	Perspective warp	DLT homography + backward mapping + bilinear interpolation	pipeline.md Stage 4
1.2.4	Shadow removal	Adaptive illumination correction + Retinex	pipeline.md Stage 5
1.2.5	Binarization	Sauvola local threshold + integral image optimization	pipeline.md Stage 6
1.2.6	Deskew	Hough transform line detection + rotation	pipeline.md Stage 7
1.2.7	Image enhancement	CLAHE + sharpen + contrast	pipeline.md Stage 5
1.2.8	Pipeline assembly	All stages connected, progress callback per stage	pipeline.md Complete assembly

Critical Algorithm: Perspective warp via SVD untuk homography matrix. Butuh implementasi DLT algorithm atau pin nalgebra crate.

// Pseudo untuk testing before optimization
// imageproc contour → corner detection → warp
let edges = robust_edge_detection(&gray);
let corners = find_document_corners(&edges)?;
let warped = perspective_warp(&original, corners);
let cleaned = remove_shadow(&warped);
let binary = binarize(&cleaned);
let final_img = deskew(&binary);

Acceptance: Image foto miring test → keluar hasil lurus bersih hitam-putih.

Effort: ~5-7 hari (ini bagian paling susah)

---

Milestone 1.3 — Next.js Frontend

Goal: User bisa upload foto, liat progress, download hasil.

Tasks:

#	Task	Detail
1.3.1	Init Next.js app	bun create next-app dengan Tailwind v4 + shadcn/ui
1.3.2	Landing page	Cards: Scan, Image, PDF — link ke masing-masing tool
1.3.3	Upload zone component	Drag & drop + file picker, validasi tipe/ukuran
1.3.4	Scanner page	/scan — upload area, tool options, progress bar
1.3.5	API upload route	Next.js API route → proxy ke Rust Gateway
1.3.6	Progress bar component	Animated bar + stage label dari WebSocket
1.3.7	Preview component	Before/after comparison slider
1.3.8	Result page	Preview + download button + file info
1.3.9	Error handling	Upload error, processing error, timeout

Acceptance: User upload foto → liat progress bar → download PDF.

Effort: ~3-4 hari

---

Milestone 1.4 — OCR + Searchable PDF

Goal: Output berupa PDF dengan hidden text layer — teks bisa di-copy, file bisa di-search.

Tasks:

#	Task	Detail
1.4.1	Install Tesseract data	Tambah tessdata di Docker image
1.4.2	OCR integration	leptess binding, set language, get text + word boxes
1.4.3	PDF generation	lopdf — page with image + invisible text layer
1.4.4	WebSocket progress	NATS consumer di Gateway → broadcast ke WS client
1.4.5	Auto-cleanup scheduler	NATS cron tiap 10 menit, hapus file expired >1 jam
1.4.6	Rate limiting	Redis sliding window per IP, per tool

Acceptance: Download PDF → buka di browser → teks bisa di-select + di-search.

Effort: ~3-4 hari

---

Phase 1 Total: ~14-19 hari kerja

---

Phase 2: Scanner Complete + Image Tools

Milestone 2.1 — Scanner Fallback + PWA

Goal: Scanner robust — kalau auto gagal, user bisa atur manual. Kamera langsung dari browser.

Tasks:

#	Task	Detail
2.1.1	Manual crop UI	Canvas: 4 draggable corners, background image
2.1.2	Fallback pipeline	Auto → gagal → manual → kirim corners ke worker
2.1.3	Camera capture	PWA: akses kamera via getUserMedia, capture frame
2.1.4	Auto-exposure helper	Tap to focus + exposure lock
2.1.5	Batch multi-page	Upload multiple photos → 1 PDF result

Effort: ~4-5 hari

---

Milestone 2.2 — Image Tools

Goal: Compress, resize, convert image langsung di browser (WASM).

Tasks:

#	Task	Detail
2.2.1	WASM image crate	Compile image crate to WASM via wasm-pack
2.2.2	Compress UI	Slider kualitas %, preview perbandingan ukuran
2.2.3	Resize UI	Input dimensi, lock aspect ratio, preview
2.2.4	Convert UI	Pilih format output, preview
2.2.5	Client-side processing	Semua image tool jalan di browser — no upload needed
2.2.6	Fallback server-side	Kalau WASM gagal/browser tua → upload ke server worker

WASM Strategy:

// apps/tools/backend/wasm/src/lib.rs
use wasm_bindgen::prelude::*;
use image::{DynamicImage, ImageFormat};
use std::io::Cursor;

#[wasm_bindgen]
pub fn compress_jpeg(bytes: &[u8], quality: u8) -> Vec<u8> {
    let img = image::load_from_memory(bytes).unwrap();
    let mut output = Cursor::new(Vec::new());
    img.write_to(&mut output, ImageFormat::Jpeg).unwrap();
    // quality compression via mozjpeg or custom
    output.into_inner()
}

Effort: ~4-5 hari

---

Milestone 2.3 — Background Removal

Goal: Hapus background foto pake AI model ONNX — jalan di Rust native.

Tasks:

#	Task	Detail
2.3.1	Download RMBG model	rmbg-1.4.onnx (atau model lebih kecil seperti u2net)
2.3.2	ONNX Runtime binding	ort crate — load model, run inference
2.3.3	Pre/post processing	Resize ke 1024x1024 → normalize → softmax → threshold
2.3.4	Mask application	Alpha channel: background transparent / warna solid
2.3.5	Image preview	Before/after dengan background removal

Model Options:

Model	Size	Quality	Notes
RMBG-1.4	~50MB	Excellent	BRIA, butuh license untuk commercial
U-2-Net	~170MB	Good	Open source, lebih besar
MODNet	~25MB	Good	Ringan, cepat
Dis_seg	~8MB	Decent	Paling kecil, cocok untuk VPS

Effort: ~3-4 hari

---

Milestone 2.4 — Batch Processing

Goal: Upload 10-20 foto sekaligus, diproses parallel, jadi 1 PDF.

Tasks:

#	Task	Detail
2.4.1	Batch upload UI	Drop zone accept multiple files, thumbnail list
2.4.2	Group job	1 group job = N individual jobs, track per-item progress
2.4.3	Rayon parallel	Worker process multiple pages in parallel
2.4.4	PDF merger	lopdf merge multiple pages → 1 document

Effort: ~3-4 hari

---

Phase 3: PDF + Video/Audio Tools

Milestone 3.1 — PDF Tools

#	Task	Detail
3.1.1	Merge PDF	Upload 2+ PDF, lopdf merge pages
3.1.2	Split PDF	Input pages "1-3,5,7-9", ekstrak + save jadi 1 file
3.1.3	Images to PDF	Upload images, sort order, jadi 1 PDF
3.1.4	PDF compress	Re-encode embedded images dengan kualitas lebih rendah
3.1.5	PDF to images	Tiap halaman → JPEG/PNG

Effort: ~4-5 hari

---

Milestone 3.2 — Video/Audio Tools

#	Task	Detail
3.2.1	Video compress	FFmpeg binding (ffmpeg-next), turunin bitrate + resolusi
3.2.2	Extract audio	MP4 → MP3 via FFmpeg
3.2.3	Trim video	Start/end time → cut segment
3.2.4	GIF maker	Video segment → GIF, atur FPS + dimensi
3.2.5	Audio convert	Format conversion via FFmpeg

Catatan: Video processing heavy — butuh dedicated worker dengan resource lebih besar. Queue priority: video jobs ke stream terpisah dengan max 1 concurrent.

Effort: ~5-7 hari

---

Milestone 3.3 — Final Polish

#	Task	Detail
3.3.1	Theme integration	Twilight Terminal theme dari hub
3.3.2	Responsive design	Mobile-first, touch-friendly crop
3.3.3	Error monitoring	Error tracking, alert kalau pipeline gagal
3.3.4	Load testing	k6: simulasi concurrent users, measure P50/P95/P99 latency
3.3.5	Dashboard integration	Link dari hub dashboard → tools stats

Effort: ~3-4 hari

---

Timeline Summary

Minggu 1:  Gateway + upload/download + edge detection + warp
Minggu 2:  Enhance + binarization + deskew + Next.js frontend
Minggu 3:  OCR + PDF + WebSocket + rate limit + cleanup
           ─── MVP LAUNCH (Document Scanner ready) ───
Minggu 4:  Manual crop fallback + PWA camera + batch
Minggu 5:  WASM image tools + compress/resize/convert
Minggu 6:  Background removal (ONNX) + PDF tools
           ─── V1 LAUNCH (Scanner + Image + PDF) ───
Minggu 7:  Video/audio tools + final polish
Minggu 8:  Load testing + bug fixes + deployment

Critical Path

Edge Detection ──▶ Corner Detection ──▶ Perspective Warp
       │                                       │
       │                              ┌────────┘
       │                              ▼
       │                       Shadow Removal ──▶ Binarization ──▶ Deskew
       │                                                            │
       │                                              ┌─────────────┘
       │                                              ▼
       │                                       OCR ──▶ PDF Gen ──▶ Output
       │
       └───(Kalau gagal)─── Manual Crop ◀── Frontend Canvas

Risks:

1. Edge detection paling rentan gagal — pipeline harus graceful fallback ke manual crop
2. Homography SVD implementasi perlu numerik stabil — test dengan extreme perspective angles
3. OCR kualitas sangat tergantung pada binarization — Sauvola parameter perlu tuning
4. WASM image processing size besar (~2MB gzipped) — perlu code splitting + lazy load