22 KiB
22 KiB
Architecture
LEGACY (2026-08-02): Dokumen plan ini ditulis saat infra masih Docker/Traefik. Produksi sekarang Caddy + Nix/systemd dengan port 4000-an. Gunakan hanya sebagai referensi historis.
System Overview
┌────────────────────────────────────────────────────────────────┐
│ BROWSER │
│ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │
│ │ Upload │ │ Camera │ │ Preview + Download │ │
│ │ (drag/drop)│ │ (PWA) │ │ (streaming) │ │
│ └─────┬──────┘ └─────┬──────┘ └───────────┬────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ WebSocket (progress: processing/step/percentage) │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────┬───────────────────────────────────┘
│ HTTPS / WSS
▼
┌────────────────────────────────────────────────────────────────┐
│ TRAEFIK (tools.asepharyana.my.id) │
│ Middleware chain: secure-headers → compress → rate-limit │
└──────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ tools-app (Next.js 16 / TypeScript) │
│ │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Pages/Routes │ │ API Routes │ │
│ │ / → home │ │ POST /api/upload ──▶ file │
│ │ /scan → scanner │ │ GET /api/job/:id ─▶ status │
│ │ /image → image │ │ WS /api/job/:id/ws ─▶ progress │
│ │ /pdf → pdf tools │ │ GET /api/download/:id ─▶ file │
│ └──────────────────┘ └─────────────────┘ │
│ │
│ Upload validation: MIME type, size limit (50MB), virus scan │
│ Temp storage bridge ke worker via HTTP/NATS │
└──────────────────┬────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌────────────────────────────────────────────────────────────────┐
│ API GATEWAY (Rust / Axum) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
│ │ Upload │ │ Job Manager │ │ Download │ │
│ │ (streaming │ │ (CRUD job │ │ (stream file, │ │
│ │ chunked) │ │ status) │ │ auto-delete) │ │
│ └──────┬───────┘ └──────┬───────┘ └────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ NATS JetStream │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ jobs │ │ jobs │ │ jobs │ │ │
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ progress │ │ progress │ │ progress │ │ │
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Cache (Redis) │ │
│ │ - Job metadata (status, progress, timestamps) │ │
│ │ - Rate limiting (sliding window per IP/tool) │ │
│ │ - Result metadata (file path, size, type) │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────┬────────────────────────────────────────────┘
│ consume NATS queue
▼
┌────────────────────────────────────────────────────────────────┐
│ WORKER POOL (Rust / Tokio + Rayon) │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ Scan Worker │ │ Image Worker │ │ PDF Worker │ │
│ │ ×4 instances │ │ ×2 instances │ │ ×2 instances│ │
│ │ │ │ │ │ │ │
│ │ 1. Load image │ │ 1. Load image │ │ 1. Load PDF │ │
│ │ 2. Edge detect │ │ 2. Compress │ │ 2. Merge/ │ │
│ │ 3. Warp │ │ /resize/ │ │ split │ │
│ │ 4. Enhance │ │ convert │ │ 3. Save │ │
│ │ 5. OCR │ │ 3. Save │ │ 4. Update │ │
│ │ 6. Gen PDF │ │ 4. Update │ │ job │ │
│ │ 7. Update job │ │ job status │ │ status │ │
│ │ └───────────────┘ └─────────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Temp Storage (filesystem volume / S3-compatible) │ │
│ │ Auto-cleanup: job TTL 1 jam, NATS cron tiap 10m │ │
│ └────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
Component Diagram
┌────────────────────────────────────────────┐
│ apps/tools │
│ │
│ ├── frontend/ │
│ │ ├── pages/ ← Next.js pages │
│ │ ├── components/ ← React components │
│ │ ├── lib/ ← utilities │
│ │ └── public/ ← static assets │
│ │ │
│ ├── backend/ ← Rust workspace │
│ │ ├── gateway/ ← Axum API server │
│ │ ├── workers/ ← Processing workers │
│ │ │ ├── scanner/ ← Document scanner │
│ │ │ ├── image/ ← Image tools │
│ │ │ └── pdf/ ← PDF tools │
│ │ └── common/ ← Shared libs │
│ │ │
│ └── Dockerfile │
└────────────────────────────────────────────┘
Data Flow (Document Scanner — Flow Lengkap)
1. User buka tools.asepharyana.my.id/scan
2. Upload foto via drag-drop atau kamera HP (PWA)
3. Next.js route handler menerima file
├─ Validasi: MIME type (image/*), max 50MB, virus header scan
└─ Upload chunked ke Gateway internal (HTTP POST)
4. Gateway menerima stream:
├─ Simpan ke temp storage
├─ Buat job record di Redis: {id, tool: "scan", status: "queued", progress: 0}
└─ Publish ke NATS: tools.scan.jobs {job_id, file_path, options}
5. Scan Worker consume dari NATS:
├─ Update Redis: status = "processing", progress = 10
├─ Load image (image-rs)
├─ Pipeline (detail di pipeline.md):
│ 1. Edge detection ──▶ progress 25
│ 2. Perspective warp ──▶ progress 40
│ 3. Shadow removal ──▶ progress 55
│ 4. Binarization ──▶ progress 70
│ 5. Contrast/sharpen ──▶ progress 80
│ 6. OCR ──▶ progress 90
│ 7. Generate PDF ──▶ progress 95
├─ Simpan file hasil ke temp storage
├─ Update Redis: status = "completed", progress = 100, result_path, ocr_text
└─ Publish ke NATS: tools.scan.progress {job_id, status, progress}
6. WebSocket handler di Gateway:
├─ Subscribe NATS topics tools.scan.progress
├─ Forward ke browser user (per-job-id filter)
└─ Browser update progress bar + preview
7. User download PDF:
├─ GET /api/download/:job_id
├─ Gateway stream file dari temp storage
└─ Browser save file
Tech Stack
Frontend (Next.js + TypeScript)
| Library | Fungsi |
|---|---|
| Next.js 16 | App router, API routes |
| shadcn/ui + Tailwind v4 | UI components |
| Framer Motion | Animasi progress, transisi |
| Canvas API | Preview crop manual, image manipulation client-side |
| WebSocket API | Real-time progress |
Backend (Rust)
| Crate | Fungsi |
|---|---|
axum |
HTTP server (Gateway) |
tokio |
Async runtime |
image |
Image I/O, resize, convert, compress |
imageproc |
Edge detection, contour, thresholding |
lopdf |
PDF generation, merge, split, compress |
leptess |
Tesseract OCR binding |
ort |
ONNX Runtime (background removal) |
async-nats |
NATS JetStream client |
deadpool-redis |
Redis connection pool |
redis |
Redis async client |
rayon |
Parallel processing (batch, pixel ops) |
serde |
Serialization |
tracing + opentelemetry |
Observability |
uuid |
Job ID generation |
Infrastructure
| Komponen | Fungsi |
|---|---|
| NATS JetStream | Job queue, progress pub/sub, scheduler |
| Redis | Job metadata, rate limiting, cache |
| PostgreSQL | Opsional — audit log, usage statistics |
| Tesseract | OCR engine (data files di Docker image) |
| Prometheus | Metrics (jobs/min, queue depth, latency per stage) |
Job Queue (NATS Streams & Consumers)
Streams
tools-scan-jobs → 1 stream, mirror to all scan workers
tools-image-jobs → 1 stream, mirror to all image workers
tools-pdf-jobs → 1 stream, mirror to all pdf workers
tools-progress → 1 stream, all progress events (key-value by job_id)
tools-scheduler → 1 stream, cron events
Subjects
tools.scan.jobs.{job_id} → job submission
tools.scan.progress.{job_id} → progress update (fan-out ke Gateway)
tools.image.jobs.{job_id} → job submission
tools.image.progress.{job_id} → progress update
tools.pdf.jobs.{job_id} → job submission
tools.pdf.progress.{job_id} → progress update
tools.scheduler.cleanup → cleanup expired files (every 10 min)
Redis Schema
job:{id} → Hash {status, tool, progress, file_path, result_path, ocr_text, created_at, ttl}
rate_limit:{ip}:{tool} → Sorted Set (sliding window)
file_meta:{hash} → String {original_name, size, mime}
Metrics (Prometheus)
| Metric | Type | Labels | Description |
|---|---|---|---|
tools_jobs_total |
Counter | tool, status |
Total jobs processed |
tools_jobs_in_flight |
Gauge | tool |
Currently processing jobs |
tools_queue_depth |
Gauge | tool |
NATS queue depth |
tools_processing_duration |
Histogram | tool, stage |
Duration per stage |
tools_file_size_bytes |
Histogram | tool |
Upload file size distribution |
tools_rate_limit_hits |
Counter | tool |
Rate limit violations |
Directory Structure
apps/tools/
├── frontend/
│ ├── src/
│ │ ├── app/
│ │ │ ├── page.tsx # Landing page
│ │ │ ├── scan/
│ │ │ │ ├── page.tsx # Scanner page
│ │ │ │ └── result/[id]/
│ │ │ │ └── page.tsx # Result page
│ │ │ ├── image/
│ │ │ │ ├── compress/page.tsx
│ │ │ │ ├── resize/page.tsx
│ │ │ │ ├── convert/page.tsx
│ │ │ │ └── remove-bg/page.tsx
│ │ │ ├── pdf/
│ │ │ │ ├── merge/page.tsx
│ │ │ │ ├── split/page.tsx
│ │ │ │ ├── images-to-pdf/page.tsx
│ │ │ │ └── compress/page.tsx
│ │ │ ├── api/
│ │ │ │ ├── upload/route.ts
│ │ │ │ ├── job/[id]/route.ts
│ │ │ │ │ └── ws/route.ts
│ │ │ │ └── download/[id]/route.ts
│ │ │ ├── layout.tsx
│ │ │ └── globals.css
│ │ ├── components/
│ │ │ ├── upload-zone.tsx # Drag & drop area
│ │ │ ├── progress-bar.tsx # WebSocket-connected progress
│ │ │ ├── preview.tsx # Before/after preview
│ │ │ ├── crop-editor.tsx # Manual corner adjustment
│ │ │ ├── tool-layout.tsx # Consistent tool page layout
│ │ │ └── camera-capture.tsx # PWA camera interface
│ │ ├── hooks/
│ │ │ ├── use-job-status.ts # WebSocket connection
│ │ │ ├── use-upload.ts # Upload with progress
│ │ │ └── use-camera.ts # Camera access
│ │ └── lib/
│ │ ├── utils.ts
│ │ └── types.ts
│ ├── next.config.ts
│ ├── package.json
│ └── tsconfig.json
│
├── backend/
│ ├── Cargo.toml
│ ├── gateway/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── routes/
│ │ │ ├── mod.rs
│ │ │ ├── upload.rs
│ │ │ ├── job.rs
│ │ │ ├── download.rs
│ │ │ └── ws.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── publisher.rs
│ │ ├── redis/
│ │ │ ├── mod.rs
│ │ │ ├── job.rs
│ │ │ └── ratelimit.rs
│ │ ├── metrics.rs
│ │ └── config.rs
│ │
│ ├── workers/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── scanner/
│ │ │ ├── mod.rs
│ │ │ ├── pipeline.rs
│ │ │ ├── edge.rs # Edge detection
│ │ │ ├── warp.rs # Perspective correction
│ │ │ ├── enhance.rs # Shadow removal, B&W, contrast
│ │ │ ├── ocr.rs # Tesseract wrapper
│ │ │ └── pdf.rs # Generate searchable PDF
│ │ ├── image/
│ │ │ ├── mod.rs
│ │ │ ├── compress.rs
│ │ │ ├── resize.rs
│ │ │ ├── convert.rs
│ │ │ └── remove_bg.rs
│ │ ├── pdf/
│ │ │ ├── mod.rs
│ │ │ ├── merge.rs
│ │ │ ├── split.rs
│ │ │ ├── extract.rs
│ │ │ └── compress.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── consumer.rs
│ │ └── config.rs
│ │
│ └── common/
│ ├── Cargo.toml
│ └── src/
│ ├── lib.rs
│ ├── types.rs # Shared types (JobStatus, Job, etc.)
│ ├── error.rs # Error types
│ └── nats.rs # NATS subject constants
│
├── Dockerfile
├── compose.yml # Local dev compose
└── README.md
API Design
Endpoints
| Method | Path | Description |
|---|---|---|
POST |
/api/upload |
Upload file, create job |
GET |
/api/job/:id |
Get job status + result metadata |
WS |
/api/job/:id/ws |
WebSocket — realtime progress |
GET |
/api/download/:id |
Download result file |
DELETE |
/api/job/:id |
Cancel job, delete files |
GET |
/health |
Health check |
Upload Request
POST /api/upload
Content-Type: multipart/form-data
{
file: <binary>,
tool: "scan" | "image-compress" | "image-resize" | "image-convert" | "remove-bg" |
"pdf-merge" | "pdf-split" | "images-to-pdf" | "pdf-compress",
options?: { // tool-specific options
quality?: 80, // compress quality
width?: 1920, // resize width
format?: "webp", // convert format
pages?: "1,3-5", // PDF split pages
dpi?: 300, // scan DPI
enhance?: true, // scan auto-enhance
ocr?: true // scan OCR
}
}
Response (202 Accepted)
{
"job_id": "uuid",
"status": "queued",
"tool": "scan",
"ws_url": "/api/job/uuid/ws",
"created_at": "2026-07-24T10:00:00Z",
"estimated_seconds": 5
}
WebSocket Messages
// Server → Client
{
"type": "progress",
"job_id": "uuid",
"status": "processing",
"progress": 45,
"stage": "warp",
"message": "Meluruskan perspektif dokumen..."
}
{
"type": "complete",
"job_id": "uuid",
"status": "completed",
"progress": 100,
"result": {
"download_url": "/api/download/uuid",
"file_name": "scan_20260724.pdf",
"file_size": 1245678,
"pages": 1,
"ocr_text": "Nama: Asep...",
"preview_url": "/api/job/uuid/preview"
}
}
{
"type": "error",
"job_id": "uuid",
"status": "failed",
"error": "Edge detection failed: cannot find document boundary"
}
Integration with Existing Portfolio
| Area | Detail |
|---|---|
| Domain | tools.asepharyana.my.id — tambah entry di infra/traefik/dynamic/apps.yaml |
| Dashboard | Link ke tools stats di dashboard hub yang sudah ada |
| Docker Compose | infra/compose/tools.yml — pola sama kaya hub.yml |
| CI/CD | Tambah service tools di docker-build-push.yml |
| Style | Ulang Twilight Terminal theme dari hub, konsisten visual branding |
| Monitoring | Reuse existing Prometheus + Grafana, tambah metrics tools |