asepharyana
68dea35d73
- Introduced a comprehensive Docker image architecture for the project, detailing multi-stage builds for Rust backend and Next.js frontend. - Added Docker Compose configuration for the tools service, including environment variables and volume management. - Documented CI/CD integration steps for Docker build and deployment workflows. - Implemented a detailed processing pipeline for document scanning, covering stages from image preprocessing to PDF generation. - Included edge case handling and performance budget for each stage of the pipeline. - Enhanced security considerations and rollback strategies for the tools service.
452 lines
22 KiB
Markdown
452 lines
22 KiB
Markdown
# Architecture
|
||
|
||
## System Overview
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ BROWSER │
|
||
│ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │
|
||
│ │ Upload │ │ Camera │ │ Preview + Download │ │
|
||
│ │ (drag/drop)│ │ (PWA) │ │ (streaming) │ │
|
||
│ └─────┬──────┘ └─────┬──────┘ └───────────┬────────────┘ │
|
||
│ │ │ │ │
|
||
│ ▼ ▼ ▼ │
|
||
│ ┌──────────────────────────────────────────────────────┐ │
|
||
│ │ WebSocket (progress: processing/step/percentage) │ │
|
||
│ └──────────────────────────────────────────────────────┘ │
|
||
└──────────────────────────┬───────────────────────────────────┘
|
||
│ HTTPS / WSS
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ TRAEFIK (tools.asepharyana.my.id) │
|
||
│ Middleware chain: secure-headers → compress → rate-limit │
|
||
└──────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ tools-app (Next.js 16 / TypeScript) │
|
||
│ │
|
||
│ ┌──────────────────┐ ┌─────────────────┐ │
|
||
│ │ Pages/Routes │ │ API Routes │ │
|
||
│ │ / → home │ │ POST /api/upload ──▶ file │
|
||
│ │ /scan → scanner │ │ GET /api/job/:id ─▶ status │
|
||
│ │ /image → image │ │ WS /api/job/:id/ws ─▶ progress │
|
||
│ │ /pdf → pdf tools │ │ GET /api/download/:id ─▶ file │
|
||
│ └──────────────────┘ └─────────────────┘ │
|
||
│ │
|
||
│ Upload validation: MIME type, size limit (50MB), virus scan │
|
||
│ Temp storage bridge ke worker via HTTP/NATS │
|
||
└──────────────────┬────────────────────────────────────────────┘
|
||
│ HTTP (internal)
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ API GATEWAY (Rust / Axum) │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
|
||
│ │ Upload │ │ Job Manager │ │ Download │ │
|
||
│ │ (streaming │ │ (CRUD job │ │ (stream file, │ │
|
||
│ │ chunked) │ │ status) │ │ auto-delete) │ │
|
||
│ └──────┬───────┘ └──────┬───────┘ └────────────────────┘ │
|
||
│ │ │ │
|
||
│ ▼ ▼ │
|
||
│ ┌────────────────────────────────────────────────────┐ │
|
||
│ │ NATS JetStream │ │
|
||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
|
||
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
|
||
│ │ │ jobs │ │ jobs │ │ jobs │ │ │
|
||
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
|
||
│ │ │ │ │ │ │
|
||
│ │ ▼ ▼ ▼ │ │
|
||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
|
||
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
|
||
│ │ │ progress │ │ progress │ │ progress │ │ │
|
||
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
|
||
│ └────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌────────────────────────────────────────────────────┐ │
|
||
│ │ Cache (Redis) │ │
|
||
│ │ - Job metadata (status, progress, timestamps) │ │
|
||
│ │ - Rate limiting (sliding window per IP/tool) │ │
|
||
│ │ - Result metadata (file path, size, type) │ │
|
||
│ └────────────────────────────────────────────────────┘ │
|
||
└──────────────────┬────────────────────────────────────────────┘
|
||
│ consume NATS queue
|
||
▼
|
||
┌────────────────────────────────────────────────────────────────┐
|
||
│ WORKER POOL (Rust / Tokio + Rayon) │
|
||
│ │
|
||
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
|
||
│ │ Scan Worker │ │ Image Worker │ │ PDF Worker │ │
|
||
│ │ ×4 instances │ │ ×2 instances │ │ ×2 instances│ │
|
||
│ │ │ │ │ │ │ │
|
||
│ │ 1. Load image │ │ 1. Load image │ │ 1. Load PDF │ │
|
||
│ │ 2. Edge detect │ │ 2. Compress │ │ 2. Merge/ │ │
|
||
│ │ 3. Warp │ │ /resize/ │ │ split │ │
|
||
│ │ 4. Enhance │ │ convert │ │ 3. Save │ │
|
||
│ │ 5. OCR │ │ 3. Save │ │ 4. Update │ │
|
||
│ │ 6. Gen PDF │ │ 4. Update │ │ job │ │
|
||
│ │ 7. Update job │ │ job status │ │ status │ │
|
||
│ │ └───────────────┘ └─────────────────┘ └──────────────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────────────┐ │
|
||
│ │ Temp Storage (filesystem volume / S3-compatible) │ │
|
||
│ │ Auto-cleanup: job TTL 1 jam, NATS cron tiap 10m │ │
|
||
│ └────────────────────────────────────────────────────┘ │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Component Diagram
|
||
|
||
```
|
||
┌────────────────────────────────────────────┐
|
||
│ apps/tools │
|
||
│ │
|
||
│ ├── frontend/ │
|
||
│ │ ├── pages/ ← Next.js pages │
|
||
│ │ ├── components/ ← React components │
|
||
│ │ ├── lib/ ← utilities │
|
||
│ │ └── public/ ← static assets │
|
||
│ │ │
|
||
│ ├── backend/ ← Rust workspace │
|
||
│ │ ├── gateway/ ← Axum API server │
|
||
│ │ ├── workers/ ← Processing workers │
|
||
│ │ │ ├── scanner/ ← Document scanner │
|
||
│ │ │ ├── image/ ← Image tools │
|
||
│ │ │ └── pdf/ ← PDF tools │
|
||
│ │ └── common/ ← Shared libs │
|
||
│ │ │
|
||
│ └── Dockerfile │
|
||
└────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Data Flow (Document Scanner — Flow Lengkap)
|
||
|
||
```
|
||
1. User buka tools.asepharyana.my.id/scan
|
||
2. Upload foto via drag-drop atau kamera HP (PWA)
|
||
3. Next.js route handler menerima file
|
||
├─ Validasi: MIME type (image/*), max 50MB, virus header scan
|
||
└─ Upload chunked ke Gateway internal (HTTP POST)
|
||
|
||
4. Gateway menerima stream:
|
||
├─ Simpan ke temp storage
|
||
├─ Buat job record di Redis: {id, tool: "scan", status: "queued", progress: 0}
|
||
└─ Publish ke NATS: tools.scan.jobs {job_id, file_path, options}
|
||
|
||
5. Scan Worker consume dari NATS:
|
||
├─ Update Redis: status = "processing", progress = 10
|
||
├─ Load image (image-rs)
|
||
├─ Pipeline (detail di pipeline.md):
|
||
│ 1. Edge detection ──▶ progress 25
|
||
│ 2. Perspective warp ──▶ progress 40
|
||
│ 3. Shadow removal ──▶ progress 55
|
||
│ 4. Binarization ──▶ progress 70
|
||
│ 5. Contrast/sharpen ──▶ progress 80
|
||
│ 6. OCR ──▶ progress 90
|
||
│ 7. Generate PDF ──▶ progress 95
|
||
├─ Simpan file hasil ke temp storage
|
||
├─ Update Redis: status = "completed", progress = 100, result_path, ocr_text
|
||
└─ Publish ke NATS: tools.scan.progress {job_id, status, progress}
|
||
|
||
6. WebSocket handler di Gateway:
|
||
├─ Subscribe NATS topics tools.scan.progress
|
||
├─ Forward ke browser user (per-job-id filter)
|
||
└─ Browser update progress bar + preview
|
||
|
||
7. User download PDF:
|
||
├─ GET /api/download/:job_id
|
||
├─ Gateway stream file dari temp storage
|
||
└─ Browser save file
|
||
```
|
||
|
||
## Tech Stack
|
||
|
||
### Frontend (Next.js + TypeScript)
|
||
|
||
| Library | Fungsi |
|
||
|---------|--------|
|
||
| Next.js 16 | App router, API routes |
|
||
| shadcn/ui + Tailwind v4 | UI components |
|
||
| Framer Motion | Animasi progress, transisi |
|
||
| Canvas API | Preview crop manual, image manipulation client-side |
|
||
| WebSocket API | Real-time progress |
|
||
|
||
### Backend (Rust)
|
||
|
||
| Crate | Fungsi |
|
||
|-------|--------|
|
||
| `axum` | HTTP server (Gateway) |
|
||
| `tokio` | Async runtime |
|
||
| `image` | Image I/O, resize, convert, compress |
|
||
| `imageproc` | Edge detection, contour, thresholding |
|
||
| `lopdf` | PDF generation, merge, split, compress |
|
||
| `leptess` | Tesseract OCR binding |
|
||
| `ort` | ONNX Runtime (background removal) |
|
||
| `async-nats` | NATS JetStream client |
|
||
| `deadpool-redis` | Redis connection pool |
|
||
| `redis` | Redis async client |
|
||
| `rayon` | Parallel processing (batch, pixel ops) |
|
||
| `serde` | Serialization |
|
||
| `tracing` + `opentelemetry` | Observability |
|
||
| `uuid` | Job ID generation |
|
||
|
||
### Infrastructure
|
||
|
||
| Komponen | Fungsi |
|
||
|----------|--------|
|
||
| NATS JetStream | Job queue, progress pub/sub, scheduler |
|
||
| Redis | Job metadata, rate limiting, cache |
|
||
| PostgreSQL | Opsional — audit log, usage statistics |
|
||
| Tesseract | OCR engine (data files di Docker image) |
|
||
| Prometheus | Metrics (jobs/min, queue depth, latency per stage) |
|
||
|
||
## Job Queue (NATS Streams & Consumers)
|
||
|
||
### Streams
|
||
|
||
```
|
||
tools-scan-jobs → 1 stream, mirror to all scan workers
|
||
tools-image-jobs → 1 stream, mirror to all image workers
|
||
tools-pdf-jobs → 1 stream, mirror to all pdf workers
|
||
tools-progress → 1 stream, all progress events (key-value by job_id)
|
||
tools-scheduler → 1 stream, cron events
|
||
```
|
||
|
||
### Subjects
|
||
|
||
```
|
||
tools.scan.jobs.{job_id} → job submission
|
||
tools.scan.progress.{job_id} → progress update (fan-out ke Gateway)
|
||
tools.image.jobs.{job_id} → job submission
|
||
tools.image.progress.{job_id} → progress update
|
||
tools.pdf.jobs.{job_id} → job submission
|
||
tools.pdf.progress.{job_id} → progress update
|
||
tools.scheduler.cleanup → cleanup expired files (every 10 min)
|
||
```
|
||
|
||
## Redis Schema
|
||
|
||
```
|
||
job:{id} → Hash {status, tool, progress, file_path, result_path, ocr_text, created_at, ttl}
|
||
rate_limit:{ip}:{tool} → Sorted Set (sliding window)
|
||
file_meta:{hash} → String {original_name, size, mime}
|
||
```
|
||
|
||
## Metrics (Prometheus)
|
||
|
||
| Metric | Type | Labels | Description |
|
||
|--------|------|--------|-------------|
|
||
| `tools_jobs_total` | Counter | `tool`, `status` | Total jobs processed |
|
||
| `tools_jobs_in_flight` | Gauge | `tool` | Currently processing jobs |
|
||
| `tools_queue_depth` | Gauge | `tool` | NATS queue depth |
|
||
| `tools_processing_duration` | Histogram | `tool`, `stage` | Duration per stage |
|
||
| `tools_file_size_bytes` | Histogram | `tool` | Upload file size distribution |
|
||
| `tools_rate_limit_hits` | Counter | `tool` | Rate limit violations |
|
||
|
||
## Directory Structure
|
||
|
||
```
|
||
apps/tools/
|
||
├── frontend/
|
||
│ ├── src/
|
||
│ │ ├── app/
|
||
│ │ │ ├── page.tsx # Landing page
|
||
│ │ │ ├── scan/
|
||
│ │ │ │ ├── page.tsx # Scanner page
|
||
│ │ │ │ └── result/[id]/
|
||
│ │ │ │ └── page.tsx # Result page
|
||
│ │ │ ├── image/
|
||
│ │ │ │ ├── compress/page.tsx
|
||
│ │ │ │ ├── resize/page.tsx
|
||
│ │ │ │ ├── convert/page.tsx
|
||
│ │ │ │ └── remove-bg/page.tsx
|
||
│ │ │ ├── pdf/
|
||
│ │ │ │ ├── merge/page.tsx
|
||
│ │ │ │ ├── split/page.tsx
|
||
│ │ │ │ ├── images-to-pdf/page.tsx
|
||
│ │ │ │ └── compress/page.tsx
|
||
│ │ │ ├── api/
|
||
│ │ │ │ ├── upload/route.ts
|
||
│ │ │ │ ├── job/[id]/route.ts
|
||
│ │ │ │ │ └── ws/route.ts
|
||
│ │ │ │ └── download/[id]/route.ts
|
||
│ │ │ ├── layout.tsx
|
||
│ │ │ └── globals.css
|
||
│ │ ├── components/
|
||
│ │ │ ├── upload-zone.tsx # Drag & drop area
|
||
│ │ │ ├── progress-bar.tsx # WebSocket-connected progress
|
||
│ │ │ ├── preview.tsx # Before/after preview
|
||
│ │ │ ├── crop-editor.tsx # Manual corner adjustment
|
||
│ │ │ ├── tool-layout.tsx # Consistent tool page layout
|
||
│ │ │ └── camera-capture.tsx # PWA camera interface
|
||
│ │ ├── hooks/
|
||
│ │ │ ├── use-job-status.ts # WebSocket connection
|
||
│ │ │ ├── use-upload.ts # Upload with progress
|
||
│ │ │ └── use-camera.ts # Camera access
|
||
│ │ └── lib/
|
||
│ │ ├── utils.ts
|
||
│ │ └── types.ts
|
||
│ ├── next.config.ts
|
||
│ ├── package.json
|
||
│ └── tsconfig.json
|
||
│
|
||
├── backend/
|
||
│ ├── Cargo.toml
|
||
│ ├── gateway/
|
||
│ │ ├── Cargo.toml
|
||
│ │ └── src/
|
||
│ │ ├── main.rs
|
||
│ │ ├── routes/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ ├── upload.rs
|
||
│ │ │ ├── job.rs
|
||
│ │ │ ├── download.rs
|
||
│ │ │ └── ws.rs
|
||
│ │ ├── nats/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ └── publisher.rs
|
||
│ │ ├── redis/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ ├── job.rs
|
||
│ │ │ └── ratelimit.rs
|
||
│ │ ├── metrics.rs
|
||
│ │ └── config.rs
|
||
│ │
|
||
│ ├── workers/
|
||
│ │ ├── Cargo.toml
|
||
│ │ └── src/
|
||
│ │ ├── main.rs
|
||
│ │ ├── scanner/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ ├── pipeline.rs
|
||
│ │ │ ├── edge.rs # Edge detection
|
||
│ │ │ ├── warp.rs # Perspective correction
|
||
│ │ │ ├── enhance.rs # Shadow removal, B&W, contrast
|
||
│ │ │ ├── ocr.rs # Tesseract wrapper
|
||
│ │ │ └── pdf.rs # Generate searchable PDF
|
||
│ │ ├── image/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ ├── compress.rs
|
||
│ │ │ ├── resize.rs
|
||
│ │ │ ├── convert.rs
|
||
│ │ │ └── remove_bg.rs
|
||
│ │ ├── pdf/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ ├── merge.rs
|
||
│ │ │ ├── split.rs
|
||
│ │ │ ├── extract.rs
|
||
│ │ │ └── compress.rs
|
||
│ │ ├── nats/
|
||
│ │ │ ├── mod.rs
|
||
│ │ │ └── consumer.rs
|
||
│ │ └── config.rs
|
||
│ │
|
||
│ └── common/
|
||
│ ├── Cargo.toml
|
||
│ └── src/
|
||
│ ├── lib.rs
|
||
│ ├── types.rs # Shared types (JobStatus, Job, etc.)
|
||
│ ├── error.rs # Error types
|
||
│ └── nats.rs # NATS subject constants
|
||
│
|
||
├── Dockerfile
|
||
├── compose.yml # Local dev compose
|
||
└── README.md
|
||
```
|
||
|
||
## API Design
|
||
|
||
### Endpoints
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `POST` | `/api/upload` | Upload file, create job |
|
||
| `GET` | `/api/job/:id` | Get job status + result metadata |
|
||
| `WS` | `/api/job/:id/ws` | WebSocket — realtime progress |
|
||
| `GET` | `/api/download/:id` | Download result file |
|
||
| `DELETE` | `/api/job/:id` | Cancel job, delete files |
|
||
| `GET` | `/health` | Health check |
|
||
|
||
### Upload Request
|
||
|
||
```
|
||
POST /api/upload
|
||
Content-Type: multipart/form-data
|
||
|
||
{
|
||
file: <binary>,
|
||
tool: "scan" | "image-compress" | "image-resize" | "image-convert" | "remove-bg" |
|
||
"pdf-merge" | "pdf-split" | "images-to-pdf" | "pdf-compress",
|
||
options?: { // tool-specific options
|
||
quality?: 80, // compress quality
|
||
width?: 1920, // resize width
|
||
format?: "webp", // convert format
|
||
pages?: "1,3-5", // PDF split pages
|
||
dpi?: 300, // scan DPI
|
||
enhance?: true, // scan auto-enhance
|
||
ocr?: true // scan OCR
|
||
}
|
||
}
|
||
```
|
||
|
||
### Response (202 Accepted)
|
||
|
||
```json
|
||
{
|
||
"job_id": "uuid",
|
||
"status": "queued",
|
||
"tool": "scan",
|
||
"ws_url": "/api/job/uuid/ws",
|
||
"created_at": "2026-07-24T10:00:00Z",
|
||
"estimated_seconds": 5
|
||
}
|
||
```
|
||
|
||
### WebSocket Messages
|
||
|
||
```json
|
||
// Server → Client
|
||
{
|
||
"type": "progress",
|
||
"job_id": "uuid",
|
||
"status": "processing",
|
||
"progress": 45,
|
||
"stage": "warp",
|
||
"message": "Meluruskan perspektif dokumen..."
|
||
}
|
||
|
||
{
|
||
"type": "complete",
|
||
"job_id": "uuid",
|
||
"status": "completed",
|
||
"progress": 100,
|
||
"result": {
|
||
"download_url": "/api/download/uuid",
|
||
"file_name": "scan_20260724.pdf",
|
||
"file_size": 1245678,
|
||
"pages": 1,
|
||
"ocr_text": "Nama: Asep...",
|
||
"preview_url": "/api/job/uuid/preview"
|
||
}
|
||
}
|
||
|
||
{
|
||
"type": "error",
|
||
"job_id": "uuid",
|
||
"status": "failed",
|
||
"error": "Edge detection failed: cannot find document boundary"
|
||
}
|
||
```
|
||
|
||
## Integration with Existing Portfolio
|
||
|
||
| Area | Detail |
|
||
|------|--------|
|
||
| **Domain** | `tools.asepharyana.my.id` — tambah entry di `infra/traefik/dynamic/apps.yaml` |
|
||
| **Dashboard** | Link ke tools stats di dashboard hub yang sudah ada |
|
||
| **Docker Compose** | `infra/compose/tools.yml` — pola sama kaya `hub.yml` |
|
||
| **CI/CD** | Tambah service `tools` di `docker-build-push.yml` |
|
||
| **Style** | Ulang Twilight Terminal theme dari hub, konsisten visual branding |
|
||
| **Monitoring** | Reuse existing Prometheus + Grafana, tambah metrics tools |
|