asepharyana-hub/docs/plan/tools/infrastructure.md

Infrastructure & Deployment

Docker Image Architecture

Project ini punya satu Docker image dengan multi-stage build. Backend Rust + Tesseract + ONNX model plus frontend Next.js.

Dockerfile Structure

# ============================================================
# Stage 1: Build Rust Backend
# ============================================================
FROM rust:1.85-slim-bookworm AS chef
RUN cargo install cargo-chef
WORKDIR /app

FROM chef AS planner
COPY backend/ .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

COPY backend/ .
RUN cargo build --release --bin gateway --bin workers

# ============================================================
# Stage 2: Build Next.js Frontend
# ============================================================
FROM oven/bun:1.3 AS frontend-builder
WORKDIR /app
COPY frontend/package.json frontend/bun.lock ./
RUN bun install --frozen-lockfile
COPY frontend/ .
RUN bun run build

# ============================================================
# Stage 3: Production Runtime
# ============================================================
FROM debian:bookworm-slim AS runtime

# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr \
    tesseract-ocr-eng \
    tesseract-ocr-ind \
    ca-certificates \
    fonts-dejavu-core \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy Rust binaries
COPY --from=builder /app/target/release/gateway /app/gateway
COPY --from=builder /app/target/release/workers /app/workers

# Copy Next.js build
COPY --from=frontend-builder /app/.next /app/.next
COPY --from=frontend-builder /app/public /app/public
COPY --from=frontend-builder /app/package.json /app/package.json
COPY --from=frontend-builder /app/node_modules /app/node_modules

# Copy ONNX model (for background removal)
COPY models/ /app/models/

# Create temp storage directory
RUN mkdir -p /data/tools && chmod 1777 /data/tools

# Environment
ENV TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
ENV TOOLS_STORAGE_PATH=/data/tools
ENV TOOLS_GATEWAY_PORT=3001
ENV TOOLS_WORKER_CONCURRENCY=4
ENV RUST_LOG=info

# Expose port
EXPOSE 3001

# Run both gateway and workers via supervisor script
COPY scripts/entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh

CMD ["/app/entrypoint.sh"]

Entrypoint Script

#!/bin/bash
# Start Gateway (Axum HTTP server)
/app/gateway &
GATEWAY_PID=$!

# Start Worker(s)
/app/workers &
WORKER_PID=$!

# Handle graceful shutdown
trap "kill $GATEWAY_PID $WORKER_PID; exit 0" SIGINT SIGTERM

# Wait for either process to exit
wait -n $GATEWAY_PID $WORKER_PID

# If one exits, kill the other
kill $GATEWAY_PID $WORKER_PID 2>/dev/null
exit 1

Image Size Estimates

Component	Size
Rust binary (gateway)	~8 MB
Rust binary (workers)	~15 MB
Next.js build	~10 MB
Tesseract + data	~25 MB
ONNX model	~50 MB
Base (Debian slim)	~80 MB
Total	~188 MB

ONNX model opsional — bisa di-download runtime daripada di-include di image.

---

Docker Compose

# infra/compose/tools.yml
services:
  tools:
    container_name: tools
    image: ghcr.io/asepharyana/asepharyana-hub/tools:sha-xxxxxxx
    restart: always
    networks:
      app-shared-net:
        aliases:
          - tools
    env_file:
      - ../../.env
    environment:
      - REDIS_URL=redis://redis:6379
      - NATS_URL=nats://nats:4222
      - TOOLS_STORAGE_PATH=/data/tools
      - TOOLS_GATEWAY_PORT=3001
      - TOOLS_WORKER_CONCURRENCY=4
      - RUST_LOG=info
    volumes:
      - tools_data:/data/tools
    ports:
      - "3001:3001"
    depends_on:
      redis:
        condition: service_started
      nats:
        condition: service_started

volumes:
  tools_data:

networks:
  app-shared-net:
    name: app-shared-net
    external: true

Environment Variables (../../.env)

# Tools
TOOLS_GATEWAY_PORT=3001
TOOLS_WORKER_CONCURRENCY=4
TOOLS_STORAGE_PATH=/data/tools
TOOLS_JOB_TTL_SECONDS=3600
TOOLS_RATE_LIMIT_PER_MINUTE=30
TOOLS_MAX_FILE_SIZE_MB=50
TOOLS_OCR_LANG=eng+ind

# Infra (reuse existing)
REDIS_URL=redis://redis:6379
NATS_URL=nats://nats:4222

---

CI/CD Integration

Docker Build Workflow

Tambah service tools di .github/workflows/docker-build-push.yml:

# Di job "changes" step "Detect changed services"
changed() {
  printf '%s\n' "$CHANGED_FILES" | grep -Eq "$1" && echo true || echo false
}
echo "tools=$(changed '^(apps/tools(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/tools\.Dockerfile$)')" >> "$GITHUB_OUTPUT"

# Di job "build" step "Set matrix"
if [ "${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then
  add_service "tools" "docker-tools" "apps/tools"
fi

# Di job "build" step "Docker metadata"
case "$SVC_NAME" in
  "tools") echo "dockerfile=infra/docker/tools.Dockerfile" >> $GITHUB_OUTPUT ;;
esac

# Di job "update-manifest"
SERVICES["tools"]="tools.yml"
PATHS["tools"]="apps/tools"

Deploy Workflow

Tambah di .github/workflows/deploy-docker.yml:

# Tidak perlu perubahan — deploy-docker.yml auto-detect compose file changes.
# Kalau compose/tools.yml berubah, service tools akan di-restart.

Service Registration (update infra/traefik/dynamic/apps.yaml)

tools:
  rule: 'Host(`tools.asepharyana.my.id`) || Host(`tools.asepharyana.web.id`)'
  entryPoints:
    - websecure
  tls: {}
  middlewares:
    - common-chain@file
  service: tools-service

# ...di bagian services:
tools-service:
  loadBalancer:
    servers:
      - url: 'http://tools:3001'

---

Monitoring

Prometheus Metrics

Tambahkan label Prometheus ke container tools:

# Di compose tools.yml
labels:
  - 'prometheus.io/scrape=true'
  - 'prometheus.io/port=3001'
  - 'prometheus.io/path=/metrics'

Dashboard Integration

Tambah card di dashboard hub yang sudah ada:

// Di dashboard hub — tambah section "Tools Usage"
// Data dari /api/dashboard → Prometheus query:
//   rate(tools_jobs_total[24h]) — jobs per tool per hari
//   sum(increase(tools_jobs_total[7d])) — total jobs minggu ini
//   tools_jobs_in_flight — current processing

---

Storage Architecture

Temp Storage

/data/tools/
├── upload/          # Uploaded files
│   └── {job_id}.{ext}
├── processing/      # Intermediate files (stage-by-stage)
│   └── {job_id}/
│       ├── 00_original.png
│       ├── 01_grayscale.png
│       ├── 02_edges.png
│       ├── 03_warped.png
│       └── ...
└── output/          # Final output
    └── {job_id}.pdf

Cleanup Strategy

Mekanisme	Timing
NATS cron job	Setiap 10 menit
Scan files >1 jam	find /data/tools -mmin +60 -delete
Redis job keys >1 jam	SCAN 0 MATCH job:* → TTL check → DEL
Storage low warning	Alert via Notification Hub (future)

---

Resource Estimation (VPS orangevps)

Current Usage

Service	CPU	RAM	Disk
Traefik	0.1	50 MB	10 MB
NATS	0.05	30 MB	10 MB
Redis	0.05	10 MB	5 MB
Dapr Placement	0.02	20 MB	5 MB
Scraper API	0.1	30 MB	50 MB
Hub	0.05	120 MB	200 MB
Jaeger	0.1	200 MB	500 MB
Prometheus	0.1	150 MB	1 GB
Node Exporter	0.02	10 MB	5 MB
OTel Collector	0.05	50 MB	10 MB
Total Current	~0.64	~670 MB	~1.8 GB

Tools Addition

Resources	Estimate	Notes
CPU	+1.0 core (burst)	Pipeline processing berat di CPU. Scoring, warp, OCR semua CPU-bound.
RAM	+300 MB	Rust binary + image processing buffers + Tesseract + ONNX
Disk	+5 GB	Temp files, bisa lebih untuk batch processing. Butuh auto-cleanup ketat.
Total After	~1.64 cores	~970 MB RAM

Catatan: Kalau VPS cuma punya 1-2 cores, processing akan antri. NATS queue handle ini. Untuk production, pastikan CPU ada >2 cores.

Scalability

VPS 1 core:
  - Scanner: ~5-8 detik per page
  - Concurrent: 1 job at a time
  - Antrian: NATS queue buffer unlimited

VPS 4+ core:
  - Scanner: ~2-3 detik per page
  - Concurrent: 4 jobs parallel (1 per worker)
  - Rayon: parallel per-page dalam batch

---

Security Considerations

Area	Mitigation
Upload validation	MIME type check (whitelist), magic bytes verification, max size 50MB
Path traversal	Job ID = UUID v4, no user-controlled filenames in storage
Command injection	No shell commands — semua processing via Rust crates, FFmpeg via crate binding
Temporary files	Auto-cleanup, random filenames, restricted permissions (0600)
Rate limiting	Redis sliding window: 30 requests/min/IP per tool, 429 response
CORS	Origin terbatas ke domain portfolio
Resource exhaustion	Max image dimension 8000px, max file count per batch 50, worker concurrency limit
OCR data	Tesseract data dari package manager, no user-trained models
ONNX model	Model dari source terpercaya, verify checksum

---

Rollback Strategy

1. Image tag: tools:sha-<short> immutable — tinggal update compose file ke tag sebelumnya
2. Data: Files auto-expire dalam 1 jam — no persistent data migration needed
3. Traefik: Cukup restart, TLS certs ga berubah
4. Monitor: Prometheus metrics akan langsung show error rate spike

---

Development Setup (Local)

Untuk development tanpa Docker:

# Terminal 1: Redis + NATS
docker compose -f infra/compose/shared.yml -f infra/compose/nats.yml up -d

# Terminal 2: Rust workers
cd apps/tools/backend
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
cargo run --bin workers

# Terminal 3: Rust gateway
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
TOOLS_STORAGE_PATH=/tmp/tools \
cargo run --bin gateway

# Terminal 4: Next.js
cd apps/tools/frontend
bun dev --port 3002

Test Pipeline Locally (tanpa NATS/Redis)

Untuk development pipeline image processing doang:

// Di workers/src/scanner/pipeline.rs — test function
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_full_pipeline() {
        let pipeline = ScanPipeline::default();
        let result = pipeline.process_sync(
            "test_images/scan_miring.jpg",
            ScanOptions { ocr: false, enhance: true }
        );
        assert!(result.is_ok());
        assert!(result.unwrap().output_path.exists());
    }

    #[test]
    fn test_edge_detection_variations() {
        // Test dengan berbagai kondisi: kertas putih, background ramai, sudut ekstrim
        for case in &["normal.jpg", "dark.jpg", "angle45.jpg", "shadow.jpg"] {
            let img = image::open(format!("test_images/{}", case)).unwrap();
            let corners = detect_corners_with_fallback(&img.grayscale().into_luma8());
            assert!(corners.is_ok(), "Failed on: {}", case);
        }
    }
}

Test images kumpulin dari foto dokumen real di berbagai kondisi — ini penting buat tuning parameter.