docs: restructure all README.md into cohesive hierarchy
- Root README redesigned as landing page with 7 sub-chapters - Each child README gets navigation header + footer linking back to root - Cross-links between Machine_Learning, ml-service, and infra READMEs - Reduced duplication: root summarizes, children provide full detail - Net -207 lines, cleaner structure Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
MythEclipse
@@ -1,6 +1,8 @@
|
|||||||
# Corn Leaf Disease Classification
|
# Pipeline Machine Learning — ZeaVis Edu
|
||||||
|
|
||||||
Pipeline lengkap untuk klasifikasi penyakit daun jagung menggunakan **EfficientNetV2B0**, mulai dari persiapan dataset, pelatihan di Google Colab, hingga ekspor model ke format **TFLite** dan **TensorFlow.js** untuk kebutuhan produksi.
|
> Panduan lengkap: preprocessing dataset, pelatihan di Google Colab, ekspor model ke TFLite, TensorFlow.js, dan ONNX.
|
||||||
|
|
||||||
|
← [Kembali ke README utama](../README.md)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -390,4 +392,8 @@ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Sesi Colab terputus saat training
|
### Sesi Colab terputus saat training
|
||||||
**Solusi:** Gunakan callback `ModelCheckpoint` di notebook untuk menyimpan checkpoint secara berkala ke Google Drive, sehingga training bisa dilanjutkan dari checkpoint terakhir tanpa mengulang dari awal.
|
**Solusi:** Gunakan callback `ModelCheckpoint` di notebook untuk menyimpan checkpoint secara berkala ke Google Drive, sehingga training bisa dilanjutkan dari checkpoint terakhir tanpa mengulang dari awal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
← [Kembali ke README utama](../README.md) • [ML Service →](../apps/ml-service/README.md)
|
||||||
|
|||||||
@@ -1,482 +1,288 @@
|
|||||||
# ZeaVis Edu
|
# ZeaVis Edu
|
||||||
|
|
||||||
ZeaVis Edu adalah aplikasi edukasi untuk membantu mengenali penyakit daun jagung melalui klasifikasi gambar berbasis machine learning. Repositori ini menggabungkan aplikasi web, API backend, layanan inferensi ML, serta pipeline pelatihan dan ekspor model EfficientNetV2B0.
|
> Klasifikasi penyakit daun jagung berbasis machine learning — dari dataset hingga aplikasi web.
|
||||||
|
|
||||||
## Fitur Utama
|
**ZeaVis Edu** adalah aplikasi edukasi pengenalan penyakit daun jagung melalui klasifikasi gambar. Repositori ini mencakup pipeline machine learning lengkap (EfficientNetV2B0), layanan inferensi Rust/ONNX Runtime, aplikasi web React, API backend Elysia, serta infrastruktur multi-VPS dengan observabilitas penuh.
|
||||||
|
|
||||||
- Aplikasi web untuk pengalaman pengguna dan interaksi edukatif.
|
---
|
||||||
- API backend untuk status layanan, integrasi data, dan komunikasi dengan layanan ML.
|
|
||||||
- ML service berbasis Rust/Axum dengan ONNX Runtime untuk inferensi penyakit daun jagung dari gambar.
|
|
||||||
- Pipeline machine learning untuk preprocessing dataset, training di Google Colab, dan ekspor model produksi.
|
|
||||||
- Dukungan Docker untuk deployment web, API, dan ML service.
|
|
||||||
- Workspace monorepo berbasis Bun dan Moon untuk menjalankan task development, typecheck, dan build secara terpusat.
|
|
||||||
|
|
||||||
## Kelas Penyakit
|
## Daftar Isi
|
||||||
|
|
||||||
Model klasifikasi menargetkan empat label berbahasa Indonesia:
|
1. [Tentang ZeaVis Edu](#1-tentang-zeavis-edu)
|
||||||
|
2. [Kelas Penyakit](#2-kelas-penyakit)
|
||||||
|
3. [Arsitektur Proyek](#3-arsitektur-proyek)
|
||||||
|
4. [Tech Stack](#4-tech-stack)
|
||||||
|
5. [Panduan Memulai](#5-panduan-memulai)
|
||||||
|
- [5.1 Instalasi Root Workspace](#51-instalasi-root-workspace)
|
||||||
|
- [5.2 Menjalankan Aplikasi Web & API](#52-menjalankan-aplikasi-web--api)
|
||||||
|
- [5.3 Pipeline Machine Learning](#53-pipeline-machine-learning)
|
||||||
|
- [5.4 ML Inference Service](#54-ml-inference-service)
|
||||||
|
- [5.5 Infrastruktur & Deployment](#55-infrastruktur--deployment)
|
||||||
|
- [5.6 Telemetry & Observabilitas](#56-telemetry--observabilitas)
|
||||||
|
6. [Dokumentasi Lengkap](#6-dokumentasi-lengkap)
|
||||||
|
7. [Troubleshooting Umum](#7-troubleshooting-umum)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Tentang ZeaVis Edu
|
||||||
|
|
||||||
|
ZeaVis Edu membantu petani, peneliti, dan pelajar mengidentifikasi penyakit daun jagung secara cepat menggunakan kamera. Pengguna cukup mengunggah foto daun jagung dan sistem akan mengklasifikasikannya ke dalam salah satu dari empat kategori kondisi daun.
|
||||||
|
|
||||||
|
### Fitur Utama
|
||||||
|
|
||||||
|
- **Aplikasi Web** — Antarmuka pengguna interaktif berbasis React + Vite
|
||||||
|
- **API Backend** — Layanan data dan autentikasi berbasis Elysia + PostgreSQL
|
||||||
|
- **ML Inference Service** — Inferensi real-time via Rust/Axum + ONNX Runtime
|
||||||
|
- **Pipeline ML** — Preprocessing dataset, pelatihan di Google Colab, ekspor model ke TFLite, TFJS, dan ONNX
|
||||||
|
- **Infrastruktur Multi-VPS** — Deployment produksi dengan Tailscale mesh VPN
|
||||||
|
- **Observabilitas** — Prometheus → ClickHouse pipeline dengan dashboard metrik
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Kelas Penyakit
|
||||||
|
|
||||||
|
Model mengklasifikasikan gambar ke dalam **4 kelas** berbahasa Indonesia:
|
||||||
|
|
||||||
| Label | Deskripsi |
|
| Label | Deskripsi |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Bercak Daun | Gray Leaf Spot |
|
| **Bercak Daun** | *Gray Leaf Spot* — bercak abu-abu memanjang |
|
||||||
| Hawar Daun | Northern/Southern Leaf Blight |
|
| **Hawar Daun** | *Northern/Southern Leaf Blight* — hawar coklat berbentuk panjang |
|
||||||
| Karat Daun | Common Rust |
|
| **Karat Daun** | *Common Rust* — bintik coklat kemerahan berbentuk pustula |
|
||||||
| Daun Sehat | Daun jagung tanpa gejala penyakit |
|
| **Daun Sehat** | Daun jagung tanpa gejala penyakit |
|
||||||
|
|
||||||
## Struktur Proyek
|
---
|
||||||
|
|
||||||
```text
|
## 3. Arsitektur Proyek
|
||||||
|
|
||||||
|
```
|
||||||
.
|
.
|
||||||
├── apps/
|
├── apps/
|
||||||
│ ├── api/ # Backend Elysia/Bun
|
│ ├── api/ # Backend Elysia/Bun + Drizzle ORM + PostgreSQL
|
||||||
│ ├── ml-service/ # Layanan inferensi Rust/Axum + ONNX Runtime
|
│ ├── ml-service/ # Layanan inferensi Rust/Axum + ONNX Runtime
|
||||||
│ └── web/ # Frontend React + Vite
|
│ └── web/ # Frontend React + Vite + Tailwind
|
||||||
├── Machine_Learning/ # Pipeline dataset, training, dan ekspor model
|
├── Machine_Learning/ # Pipeline dataset, training, dan ekspor model
|
||||||
|
│ └── README.md # ⤷ Panduan lengkap pipeline ML
|
||||||
|
├── infra/
|
||||||
|
│ └── README.md # ⤷ Panduan deployment multi-VPS
|
||||||
├── packages/
|
├── packages/
|
||||||
│ └── shared/ # Tipe dan utilitas bersama TypeScript
|
│ └── shared/ # Tipe dan utilitas bersama TypeScript
|
||||||
├── docker-compose.yml # Konfigurasi deployment container
|
├── telemetry/ # Submodule — Prometheus → ClickHouse pipeline
|
||||||
├── package.json # Script dan workspace root Bun
|
├── docker-compose.yml # Konfigurasi deployment container
|
||||||
└── README.md # Dokumentasi utama proyek
|
├── package.json # Root workspace Bun + Moon
|
||||||
|
└── README.md # ⤷ Anda di sini
|
||||||
```
|
```
|
||||||
|
|
||||||
## Tech Stack
|
| Komponen | Teknologi | Dokumentasi |
|
||||||
|
|---|---|---|
|
||||||
|
| Web Frontend | React, Vite, Tailwind, Zustand | `apps/web/` |
|
||||||
|
| API Backend | Bun, Elysia, Drizzle ORM, PostgreSQL | `apps/api/` |
|
||||||
|
| ML Inference | Rust, Axum, ONNX Runtime | [`apps/ml-service/README.md`](apps/ml-service/README.md) |
|
||||||
|
| ML Pipeline | Python, TensorFlow/Keras, EfficientNetV2B0 | [`Machine_Learning/README.md`](Machine_Learning/README.md) |
|
||||||
|
| Infrastruktur | Docker, Coolify, Traefik, Tailscale | [`infra/README.md`](infra/README.md) |
|
||||||
|
| Telemetry | Prometheus, ClickHouse, Vector, Vue 3 | `telemetry/` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Tech Stack
|
||||||
|
|
||||||
### Frontend
|
### Frontend
|
||||||
|
React • Vite • TypeScript • React Router • TanStack Query • Zustand • Tailwind CSS
|
||||||
- React
|
|
||||||
- Vite
|
|
||||||
- TypeScript
|
|
||||||
- React Router
|
|
||||||
- TanStack Query
|
|
||||||
- Zustand
|
|
||||||
- Tailwind CSS
|
|
||||||
|
|
||||||
### Backend API
|
### Backend API
|
||||||
|
Bun • Elysia • Drizzle ORM • PostgreSQL
|
||||||
- Bun
|
|
||||||
- Elysia
|
|
||||||
- Drizzle ORM
|
|
||||||
- PostgreSQL
|
|
||||||
|
|
||||||
### Machine Learning
|
### Machine Learning
|
||||||
|
Python • TensorFlow/Keras • EfficientNetV2B0 • Rust • Axum • ONNX Runtime • TFLite • TensorFlow.js
|
||||||
- Python (preprocessing, training, export)
|
|
||||||
- TensorFlow/Keras
|
|
||||||
- EfficientNetV2B0
|
|
||||||
- Rust
|
|
||||||
- Axum
|
|
||||||
- ONNX Runtime
|
|
||||||
- TFLite
|
|
||||||
- TensorFlow.js
|
|
||||||
|
|
||||||
### Tooling & Deployment
|
### Tooling & Deployment
|
||||||
|
Bun workspaces • Moon task runner • Docker • Docker Compose • Coolify • Traefik
|
||||||
|
|
||||||
- Bun workspaces
|
### Telemetry & Observabilitas
|
||||||
- Moon task runner
|
Prometheus • Metric Ingester (Go) • Vector • ClickHouse • Query Proxy (Go) • Telemetry UI (Vue 3)
|
||||||
- Docker
|
|
||||||
- Docker Compose
|
|
||||||
- GitHub Container Registry
|
|
||||||
- Traefik labels untuk routing deployment
|
|
||||||
|
|
||||||
### Telemetry & Observability
|
---
|
||||||
|
|
||||||
- Prometheus — metric scraping & remote_write
|
## 5. Panduan Memulai
|
||||||
- Metric Ingester (Go) — enrichment, filtering, aggregation
|
|
||||||
- Vector — buffering & backpressure
|
|
||||||
- ClickHouse — columnar analytical storage
|
|
||||||
- Query Proxy (Go) — read-only SQL proxy
|
|
||||||
- Telemetry UI (Vue 3) — metrics dashboard
|
|
||||||
- Semua service ZeaVis Edu (web, api, ml-service) mengekspos metrik Prometheus di `/metrics`
|
|
||||||
- Client-side Web Vitals (CLS, FCP, INP, LCP, TTFB) dikumpulkan di frontend
|
|
||||||
|
|
||||||
## Prasyarat
|
### Prasyarat
|
||||||
|
|
||||||
Untuk menjalankan seluruh project secara lokal, siapkan:
|
- **Bun** — untuk workspace TypeScript
|
||||||
|
- **Python 3.9–3.11** — untuk pipeline ML
|
||||||
|
- **Rust & Cargo** — untuk `apps/ml-service`
|
||||||
|
- **Docker & Docker Compose** — untuk deployment dan telemetry
|
||||||
|
- **PostgreSQL** — untuk backend API
|
||||||
|
|
||||||
- Bun
|
### 5.1 Instalasi Root Workspace
|
||||||
- Python 3.9–3.11 untuk pipeline ML
|
|
||||||
- Rust dan Cargo untuk `apps/ml-service`
|
|
||||||
- Docker dan Docker Compose jika ingin menjalankan/deploy via container
|
|
||||||
- PostgreSQL jika fitur backend yang membutuhkan database digunakan
|
|
||||||
- File model `Machine_Learning/model/model.onnx` untuk inferensi ML lokal
|
|
||||||
|
|
||||||
## Instalasi Root Workspace
|
|
||||||
|
|
||||||
Jalankan dari root repository:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Clone repositori
|
||||||
|
git clone https://github.com/mytheclipse/zeavis-edu.git
|
||||||
|
cd zeavis-edu
|
||||||
|
|
||||||
|
# Instal dependensi TypeScript workspace
|
||||||
bun install
|
bun install
|
||||||
```
|
```
|
||||||
|
|
||||||
## Menjalankan Project Lokal
|
### 5.2 Menjalankan Aplikasi Web & API
|
||||||
|
|
||||||
### Menjalankan Semua Task Development
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Semua task development (web + api)
|
||||||
bun run dev
|
bun run dev
|
||||||
|
|
||||||
|
# Atau jalankan terpisah:
|
||||||
|
cd apps/web && bun run dev # Frontend — Vite dev server
|
||||||
|
cd apps/api && bun run start # API backend
|
||||||
```
|
```
|
||||||
|
|
||||||
Script ini menjalankan task `dev` melalui Moon untuk workspace yang tersedia.
|
**Environment variables** yang perlu disiapkan di root `.env`:
|
||||||
|
|
||||||
### Type Check
|
| Variable | Keterangan |
|
||||||
|
|---|---|
|
||||||
|
| `DATABASE_URL` | URL koneksi PostgreSQL |
|
||||||
|
| `SESSION_SECRET` | Secret untuk session auth |
|
||||||
|
| `WEB_APP_URL` | URL frontend (untuk CORS) |
|
||||||
|
| `ML_SERVICE_URL` | URL layanan inferensi ML |
|
||||||
|
|
||||||
```bash
|
### 5.3 Pipeline Machine Learning
|
||||||
bun run typecheck
|
|
||||||
```
|
|
||||||
|
|
||||||
### Build Produksi
|
Pipeline lengkap preprocessing dataset, pelatihan model di Google Colab, dan ekspor ke berbagai format produksi.
|
||||||
|
|
||||||
```bash
|
> 📖 **Panduan lengkap:** [`Machine_Learning/README.md`](Machine_Learning/README.md)
|
||||||
bun run build
|
|
||||||
```
|
|
||||||
|
|
||||||
## Menjalankan Service Secara Terpisah
|
**Ringkasan alur:**
|
||||||
|
|
||||||
### Web App
|
1. Unduh 3 file ZIP dataset ke `Machine_Learning/`
|
||||||
|
2. Jalankan `python preprocessing.py` untuk menggabungkan dataset
|
||||||
|
3. Upload `dataset.zip` ke Google Drive
|
||||||
|
4. Jalankan `notebook.ipynb` di Google Colab (GPU T4)
|
||||||
|
5. Download `best_model.keras` hasil training
|
||||||
|
6. Ekspor model: `python save_model.py` → TFLite + SavedModel
|
||||||
|
7. Konversi ke TFJS dan ONNX
|
||||||
|
|
||||||
```bash
|
**Output pipeline:**
|
||||||
cd apps/web
|
|
||||||
bun run dev
|
|
||||||
```
|
|
||||||
|
|
||||||
Secara default Vite akan menjalankan server development dan menampilkan URL lokal di terminal.
|
| Path | Format | Kegunaan |
|
||||||
|
|---|---|---|
|
||||||
|
| `Machine_Learning/model/saved_model/` | TensorFlow SavedModel | Jembatan konversi |
|
||||||
|
| `Machine_Learning/model/model.tflite` | TFLite | Mobile (Android/iOS) |
|
||||||
|
| `Machine_Learning/model/model.onnx` | ONNX | Rust inference service |
|
||||||
|
| `Machine_Learning/model/tfjs_model/` | TensorFlow.js | Web browser |
|
||||||
|
|
||||||
### API Backend
|
### 5.4 ML Inference Service
|
||||||
|
|
||||||
```bash
|
Layanan inferensi berbasis Rust/Axum dengan ONNX Runtime untuk prediksi real-time.
|
||||||
cd apps/api
|
|
||||||
bun run start
|
|
||||||
```
|
|
||||||
|
|
||||||
API membaca konfigurasi dari file `.env` di root repository melalui script Bun.
|
> 📖 **Panduan lengkap:** [`apps/ml-service/README.md`](apps/ml-service/README.md)
|
||||||
|
|
||||||
Script lain yang tersedia:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bun run db:generate
|
|
||||||
bun run db:migrate
|
|
||||||
bun run db:seed
|
|
||||||
bun run typecheck
|
|
||||||
```
|
|
||||||
|
|
||||||
### ML Service
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd apps/ml-service
|
cd apps/ml-service
|
||||||
cargo run
|
cargo run # Default port 8000
|
||||||
|
|
||||||
|
# Atau dengan konfigurasi custom:
|
||||||
|
MODEL_PATH=/path/to/model.onnx ML_SERVICE_PORT=9000 cargo run
|
||||||
```
|
```
|
||||||
|
|
||||||
Default path model adalah:
|
**Endpoint utama:**
|
||||||
|
|
||||||
```text
|
| Method | Path | Deskripsi |
|
||||||
../../Machine_Learning/model/model.onnx
|
|
||||||
```
|
|
||||||
|
|
||||||
Jika model berada di lokasi lain, gunakan environment variable `MODEL_PATH`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
MODEL_PATH=/path/to/model.onnx cargo run
|
|
||||||
```
|
|
||||||
|
|
||||||
**Port Configuration:**
|
|
||||||
|
|
||||||
- **Default (tanpa .env):** Service mendengarkan di `http://localhost:8000`
|
|
||||||
- **Local development (dengan .env.example):** Service mendengarkan di `http://localhost:8001`
|
|
||||||
```bash
|
|
||||||
cd apps/ml-service
|
|
||||||
source .env.example
|
|
||||||
cargo run
|
|
||||||
```
|
|
||||||
- **Docker container:** Service mendengarkan di port `8000`
|
|
||||||
|
|
||||||
Lihat `apps/ml-service/README.md` untuk detail lengkap tentang konfigurasi port dan contoh curl.
|
|
||||||
|
|
||||||
## Docker Deployment
|
|
||||||
|
|
||||||
File `docker-compose.yml` di root menyiapkan tiga service produksi:
|
|
||||||
|
|
||||||
- `web` untuk frontend
|
|
||||||
- `api` untuk backend
|
|
||||||
- `ml` untuk layanan inferensi machine learning
|
|
||||||
|
|
||||||
Konfigurasi compose menggunakan image dari GitHub Container Registry:
|
|
||||||
|
|
||||||
```text
|
|
||||||
ghcr.io/${GITHUB_REPOSITORY:-mytheclipse/zeavis-edu}/web:main
|
|
||||||
ghcr.io/${GITHUB_REPOSITORY:-mytheclipse/zeavis-edu}/api:main
|
|
||||||
ghcr.io/${GITHUB_REPOSITORY:-mytheclipse/zeavis-edu}/ml:main
|
|
||||||
```
|
|
||||||
|
|
||||||
Compose juga mengasumsikan network eksternal bernama `app-shared-net` dan routing Traefik untuk domain produksi. Service `ml` berjalan pada port `8000` di dalam container.
|
|
||||||
|
|
||||||
Contoh menjalankan compose setelah environment dan network siap:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
## Telemetry Stack
|
|
||||||
|
|
||||||
Proyek ini menyertakan pipeline telemetry metric sebagai git submodule di `telemetry/`. Pipeline mengalirkan metrik dari seluruh service ZeaVis Edu ke ClickHouse untuk analisis dan visualisasi jangka panjang.
|
|
||||||
|
|
||||||
### Arsitektur (Production)
|
|
||||||
|
|
||||||
Di production, aplikasi dan telemetry berjalan di **VPS terpisah** dan terhubung via **Tailscale** (mesh VPN). Prometheus di VPS telemetry melakukan scrape ke service ZeaVis Edu melalui IP Tailscale masing-masing.
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
subgraph VPS1["VPS — ZeaVis Edu (App)"]
|
|
||||||
W[Web / React<br/>api-zeavisedu.asepharyana.id]
|
|
||||||
A[API / Elysia<br/>:3000]
|
|
||||||
M[ML Service / Axum<br/>:8000]
|
|
||||||
end
|
|
||||||
|
|
||||||
subgraph VPS2["VPS — Telemetry Stack"]
|
|
||||||
P[Prometheus<br/>:9090]
|
|
||||||
MI[Metric Ingester<br/>:9091]
|
|
||||||
V[Vector<br/>:9001]
|
|
||||||
CH[ClickHouse<br/>:8123]
|
|
||||||
QP[Query Proxy<br/>:9092]
|
|
||||||
TUI[Telemetry UI<br/>:8181]
|
|
||||||
end
|
|
||||||
|
|
||||||
P -.->|"scrape via Tailscale IP<br/>100.x.x.a:3000/metrics"| A
|
|
||||||
P -.->|"scrape via Tailscale IP<br/>100.x.x.a:8000/metrics"| M
|
|
||||||
P -->|remote_write| MI
|
|
||||||
MI --> V
|
|
||||||
V --> CH
|
|
||||||
QP --> CH
|
|
||||||
TUI --> QP
|
|
||||||
```
|
|
||||||
|
|
||||||
Setiap service ZeaVis Edu mengekspos endpoint `/metrics` dalam format Prometheus text:
|
|
||||||
|
|
||||||
| Service | Endpoint | Port (lokal) |
|
|
||||||
|-----------------------|--------------------|--------------|
|
|
||||||
| Web (Vite dev) | `GET /metrics` | 5173 |
|
|
||||||
| API (Elysia) | `GET /metrics` | 3000 |
|
|
||||||
| ML Service (Axum) | `GET /metrics` | 8000 |
|
|
||||||
|
|
||||||
Prometheus di VPS telemetry melakukan **scrape langsung** ke API dan ML service melalui IP Tailscale mereka, bukan melalui domain publik. Konfigurasi target ada di `telemetry/prometheus/targets/zeavis-edu.json` — isi dengan IP Tailscale dari service yang dituju.
|
|
||||||
|
|
||||||
Lihat [`METRICS.md`](./METRICS.md) untuk daftar lengkap metrik yang diekspos.
|
|
||||||
|
|
||||||
### Service Telemetry
|
|
||||||
|
|
||||||
| # | Service | Peran | Port |
|
|
||||||
|---|---------|------|------|
|
|
||||||
| 1 | **Prometheus** | Metric scraping & remote_write | 9090 |
|
|
||||||
| 2 | **Metric Ingester** | Enrichment, filtering, aggregation | 9091 |
|
|
||||||
| 3 | **Vector** | Buffering, backpressure, retry | 9001 |
|
|
||||||
| 4 | **ClickHouse** | Columnar analytical storage | 8123 / 9000 |
|
|
||||||
| 5 | **Query Proxy** | Read-only SQL proxy, tenant isolation | 9092 |
|
|
||||||
| 6 | **Telemetry UI** | Vue 3 metrics dashboard | 8181 |
|
|
||||||
|
|
||||||
### Arsitektur (Local Dev)
|
|
||||||
|
|
||||||
Untuk development lokal di satu mesin, telemetry dan app bisa jalan bareng di satu Docker host. Prometheus bisa scrape service lewat Docker network yang sama.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Setup network
|
|
||||||
docker network create app-shared-net
|
|
||||||
|
|
||||||
# Build & start telemetry (dengan network sharing)
|
|
||||||
make telemetry-up-local
|
|
||||||
```
|
|
||||||
|
|
||||||
### Menjalankan Telemetry Stack
|
|
||||||
|
|
||||||
Semua operasi telemetry dijalankan dari **root proyek** melalui Makefile:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build komponen telemetry (metric-ingester + telemetry-ui)
|
|
||||||
make telemetry-build
|
|
||||||
|
|
||||||
# Start semua service telemetry (mode produksi, via Tailscale)
|
|
||||||
make telemetry-up
|
|
||||||
|
|
||||||
# Start semua service telemetry (mode lokal — port langsung terbuka)
|
|
||||||
make telemetry-up-local
|
|
||||||
|
|
||||||
# Cek status kesehatan semua service
|
|
||||||
make telemetry-status
|
|
||||||
|
|
||||||
# Lihat log (semua service, atau filter dengan s=)
|
|
||||||
make telemetry-logs
|
|
||||||
make telemetry-logs s=metric-ingester
|
|
||||||
|
|
||||||
# Restart service tertentu
|
|
||||||
make telemetry-restart s=prometheus
|
|
||||||
|
|
||||||
# Kirim test metric
|
|
||||||
make telemetry-test-metric
|
|
||||||
|
|
||||||
# Stop semua service
|
|
||||||
make telemetry-down
|
|
||||||
```
|
|
||||||
|
|
||||||
Untuk development lokal:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Setup network jika belum ada
|
|
||||||
docker network create telemetry-net
|
|
||||||
docker network create app-shared-net
|
|
||||||
|
|
||||||
# Build & start
|
|
||||||
make telemetry-build
|
|
||||||
make telemetry-up-local
|
|
||||||
|
|
||||||
# Buka dashboard di http://localhost:8181
|
|
||||||
```
|
|
||||||
|
|
||||||
### Prometheus Auto-Discovery
|
|
||||||
|
|
||||||
Prometheus menggunakan `file_sd_configs` untuk menemukan target secara dinamis. Cukup letakkan file JSON di `telemetry/prometheus/targets/` dan Prometheus akan otomatis mendeteksinya dalam 15 detik — tanpa restart.
|
|
||||||
|
|
||||||
File template sudah tersedia di [`telemetry/prometheus/targets/zeavis-edu.json`](telemetry/prometheus/targets/zeavis-edu.json). **Sebelum production, isi `__CHANGE_ME__` dengan IP Tailscale masing-masing service:**
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{ "targets": ["100.x.x.a:3000"], "labels": { "service": "zeavis-api", "component": "backend", "env": "production" } },
|
|
||||||
{ "targets": ["100.x.x.a:8000"], "labels": { "service": "zeavis-ml", "component": "inference", "env": "production" } }
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Catatan:** Aplikasi ZeaVis Edu mengekspose port Docker-nya (`:3000`, `:8000`) langsung ke host via `docker-compose.yml`. Pastikan port-port tersebut terbuka di network Tailscale (biasanya iptables Tailscale mengizinkan koneksi ke port localhost).
|
|
||||||
|
|
||||||
### Environment Variables Telemetry
|
|
||||||
|
|
||||||
| Variable | Default | Deskripsi |
|
|
||||||
|----------|---------|-----------|
|
|
||||||
| `CLICKHOUSE_USER` | `telemetry` | User ClickHouse |
|
|
||||||
| `CLICKHOUSE_PASSWORD` | `telemetry` | Password ClickHouse |
|
|
||||||
|
|
||||||
## Workflow Machine Learning
|
|
||||||
|
|
||||||
Detail lengkap tersedia di [`Machine_Learning/README.md`](Machine_Learning/README.md). Ringkasnya:
|
|
||||||
|
|
||||||
1. Unduh `dataset_1.zip`, `dataset_2.zip`, dan `dataset_3.zip` lalu letakkan di `Machine_Learning/`.
|
|
||||||
2. Jalankan preprocessing lokal:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd Machine_Learning
|
|
||||||
python preprocessing.py
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Upload `dataset.zip` ke Google Drive.
|
|
||||||
4. Jalankan `notebook.ipynb` di Google Colab dengan GPU.
|
|
||||||
5. Download model terbaik sebagai `best_model/best_model.keras`.
|
|
||||||
6. Ekspor model produksi:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python save_model.py
|
|
||||||
```
|
|
||||||
|
|
||||||
7. Konversi TensorFlow.js via CLI:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
|
|
||||||
tensorflowjs_converter \
|
|
||||||
--input_format=tf_saved_model \
|
|
||||||
--output_format=tfjs_graph_model \
|
|
||||||
--signature_name=serving_default \
|
|
||||||
--saved_model_tags=serve \
|
|
||||||
model/saved_model \
|
|
||||||
model/tfjs_model
|
|
||||||
```
|
|
||||||
|
|
||||||
Output utama pipeline ML:
|
|
||||||
|
|
||||||
| Path | Kegunaan |
|
|
||||||
|---|---|
|
|
||||||
| `Machine_Learning/dataset.zip` | Dataset siap upload ke Colab |
|
|
||||||
| `Machine_Learning/best_model/best_model.keras` | Model Keras hasil training |
|
|
||||||
| `Machine_Learning/model/saved_model/` | TensorFlow SavedModel |
|
|
||||||
| `Machine_Learning/model/model.tflite` | Model untuk mobile/TFLite |
|
|
||||||
| `Machine_Learning/model/model.onnx` | Model untuk Rust ONNX Runtime |
|
|
||||||
| `Machine_Learning/model/tfjs_model/` | Model untuk TensorFlow.js |
|
|
||||||
|
|
||||||
## Artifact Lokal dan Generated Files
|
|
||||||
|
|
||||||
Beberapa file tidak tersedia di fresh clone karena berukuran besar, dihasilkan lokal, atau berasal dari sumber eksternal:
|
|
||||||
|
|
||||||
- `Machine_Learning/dataset_1.zip`
|
|
||||||
- `Machine_Learning/dataset_2.zip`
|
|
||||||
- `Machine_Learning/dataset_3.zip`
|
|
||||||
- `Machine_Learning/dataset/`
|
|
||||||
- `Machine_Learning/dataset.zip`
|
|
||||||
- `Machine_Learning/best_model/best_model.keras`
|
|
||||||
- `Machine_Learning/model/saved_model/`
|
|
||||||
- `Machine_Learning/model/model.tflite`
|
|
||||||
- `Machine_Learning/model/model.onnx`
|
|
||||||
- `Machine_Learning/model/tfjs_model/`
|
|
||||||
|
|
||||||
## Environment Variable Penting
|
|
||||||
|
|
||||||
| Variable | Digunakan oleh | Keterangan |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `DATABASE_URL` | API | URL koneksi PostgreSQL untuk Drizzle |
|
| `GET` | `/health` | Health check + status model |
|
||||||
| `API_PORT` | API | Port backend produksi |
|
| `GET` | `/metadata` | Info model, label, versi |
|
||||||
| `WEB_APP_URL` | API | URL frontend untuk konfigurasi CORS/integrasi |
|
| `POST` | `/predict` | Klasifikasi gambar (multipart upload) |
|
||||||
| `ML_SERVICE_URL` | API | URL layanan ML |
|
|
||||||
| `MODEL_PATH` | ML Service | Lokasi file model ONNX, default `../../Machine_Learning/model/model.onnx` |
|
|
||||||
| `MODEL_INPUT_SIZE` | ML Service | Ukuran input model, default produksi `224` |
|
|
||||||
|
|
||||||
## Troubleshooting
|
### 5.5 Infrastruktur & Deployment
|
||||||
|
|
||||||
### `bun run dev` gagal karena dependency belum tersedia
|
> 📖 **Panduan lengkap:** [`infra/README.md`](infra/README.md)
|
||||||
|
|
||||||
Jalankan ulang instalasi dari root repository:
|
Arsitektur produksi berjalan di **dua VPS terpisah** yang terhubung via **Tailscale** mesh VPN:
|
||||||
|
|
||||||
|
| VPS | Hostname | Peran |
|
||||||
|
|---|---|---|
|
||||||
|
| **App VPS** | `imrnes` (Arch Linux) | Web, API, ML Service |
|
||||||
|
| **Telemetry VPS** | `orange` (Ubuntu) | Prometheus, ClickHouse, Telemetry UI |
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bun install
|
# Deploy app services via Docker Compose
|
||||||
|
docker compose up -d
|
||||||
|
|
||||||
|
# Deploy telemetry stack
|
||||||
|
make telemetry-up
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.6 Telemetry & Observabilitas
|
||||||
|
|
||||||
|
Pipeline metrik lengkap: **Prometheus → Metric Ingester → Vector → ClickHouse → Telemetry UI**.
|
||||||
|
|
||||||
|
Setiap service ZeaVis Edu mengekspos endpoint `GET /metrics` dalam format Prometheus. Prometheus di Telemetry VPS melakukan scrape melalui IP Tailscale.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make telemetry-up # Mode produksi
|
||||||
|
make telemetry-up-local # Mode development lokal
|
||||||
|
make telemetry-status # Cek kesehatan
|
||||||
|
make telemetry-logs # Lihat log
|
||||||
|
```
|
||||||
|
|
||||||
|
| Service | Port | Peran |
|
||||||
|
|---|---|---|
|
||||||
|
| Prometheus | 9090 | Scraping & remote_write |
|
||||||
|
| Metric Ingester | 9091 | Enrichment & filtering |
|
||||||
|
| Vector | 9001 | Buffering |
|
||||||
|
| ClickHouse | 8123 | Penyimpanan analitik |
|
||||||
|
| Query Proxy | 9092 | Read-only SQL proxy |
|
||||||
|
| Telemetry UI | 8181 | Dashboard metrik |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Dokumentasi Lengkap
|
||||||
|
|
||||||
|
| Dokumen | Isi |
|
||||||
|
|---|---|
|
||||||
|
| [`Machine_Learning/README.md`](Machine_Learning/README.md) | Panduan lengkap pipeline ML — preprocessing, training Colab, ekspor model TFLite/TFJS/ONNX |
|
||||||
|
| [`apps/ml-service/README.md`](apps/ml-service/README.md) | ML inference service — setup, endpoint API, konfigurasi, troubleshooting |
|
||||||
|
| [`infra/README.md`](infra/README.md) | Arsitektur multi-VPS — diagram, GitHub Secrets, port, metrics flow |
|
||||||
|
| [`METRICS.md`](METRICS.md) | Daftar lengkap metrik Prometheus yang diekspos |
|
||||||
|
| `telemetry/` (submodule) | Source code telemetry stack |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Troubleshooting Umum
|
||||||
|
|
||||||
|
### `bun install` gagal
|
||||||
|
Pastikan Bun versi terbaru terinstal:
|
||||||
|
```bash
|
||||||
|
bun --version # minimal 1.x
|
||||||
```
|
```
|
||||||
|
|
||||||
### API membutuhkan database
|
### API membutuhkan database
|
||||||
|
Pastikan `DATABASE_URL` tersedia di root `.env` dan PostgreSQL dapat diakses.
|
||||||
Pastikan `DATABASE_URL` tersedia di `.env` root dan PostgreSQL dapat diakses oleh aplikasi.
|
|
||||||
|
|
||||||
### ML service gagal memuat model
|
### ML service gagal memuat model
|
||||||
|
Pastikan `Machine_Learning/model/model.onnx` sudah dibuat:
|
||||||
Pastikan file model tersedia di path default:
|
|
||||||
|
|
||||||
```text
|
|
||||||
Machine_Learning/model/model.onnx
|
|
||||||
```
|
|
||||||
|
|
||||||
Atau set path khusus:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
MODEL_PATH=/path/to/model.onnx cargo run
|
ls -la Machine_Learning/model/model.onnx
|
||||||
```
|
```
|
||||||
|
Jika belum, jalankan pipeline ML terlebih dahulu — lihat [`Machine_Learning/README.md`](Machine_Learning/README.md).
|
||||||
|
|
||||||
### Docker Compose gagal karena network tidak ditemukan
|
### Docker Compose gagal — network tidak ditemukan
|
||||||
|
|
||||||
`docker-compose.yml` menggunakan network eksternal `app-shared-net`. Buat network tersebut jika belum ada:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker network create app-shared-net
|
docker network create app-shared-net
|
||||||
```
|
```
|
||||||
|
|
||||||
### Konversi TensorFlow.js gagal karena konflik protobuf
|
### Konversi TensorFlow.js gagal (konflik protobuf)
|
||||||
|
|
||||||
Jalankan konversi melalui CLI dan set environment variable berikut:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
|
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Pengembangan
|
## Pengembangan
|
||||||
|
|
||||||
Alur umum pengembangan:
|
1. `bun install`
|
||||||
|
2. Jalankan service yang dibutuhkan secara lokal
|
||||||
|
3. `bun run typecheck` sebelum commit
|
||||||
|
4. `bun run build` untuk verifikasi build produksi
|
||||||
|
5. Ikuti dokumentasi detail di masing-masing README untuk perubahan spesifik
|
||||||
|
|
||||||
1. Install dependency dengan `bun install`.
|
---
|
||||||
2. Jalankan service yang dibutuhkan secara lokal.
|
|
||||||
3. Jalankan `bun run typecheck` sebelum membuat commit.
|
|
||||||
4. Jalankan `bun run build` untuk memverifikasi build produksi.
|
|
||||||
5. Untuk perubahan ML, ikuti dokumentasi detail di `Machine_Learning/README.md`.
|
|
||||||
6. Untuk perubahan ML service, cek juga `apps/ml-service/README.md`.
|
|
||||||
|
|
||||||
## Dokumentasi Terkait
|
> 🧬 **ZeaVis Edu** — Memberdayakan pertanian presisi melalui machine learning
|
||||||
|
|
||||||
- [`Machine_Learning/README.md`](Machine_Learning/README.md) — panduan lengkap dataset, training, dan ekspor model.
|
|
||||||
- [`apps/ml-service/README.md`](apps/ml-service/README.md) — panduan menjalankan dan memverifikasi layanan inferensi ML.
|
|
||||||
|
|||||||
+97
-163
@@ -1,8 +1,25 @@
|
|||||||
# ML Service — Rust Axum ONNX Runtime
|
# ML Inference Service — ZeaVis Edu
|
||||||
|
|
||||||
Layanan inferensi machine learning berbasis Rust dengan Axum web framework dan ONNX Runtime untuk klasifikasi penyakit daun jagung. Service ini menyediakan endpoint HTTP untuk prediksi real-time dengan performa tinggi dan konsumsi resource minimal.
|
> Layanan inferensi machine learning berbasis Rust/Axum + ONNX Runtime untuk klasifikasi penyakit daun jagung.
|
||||||
|
|
||||||
## Fitur
|
← [Kembali ke README utama](../../README.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Daftar Isi
|
||||||
|
|
||||||
|
1. [Fitur](#1-fitur)
|
||||||
|
2. [Prasyarat & Instalasi](#2-prasyarat--instalasi)
|
||||||
|
3. [Menjalankan Service](#3-menjalankan-service)
|
||||||
|
4. [Environment Variables](#4-environment-variables)
|
||||||
|
5. [Endpoint API](#5-endpoint-api)
|
||||||
|
6. [Verifikasi & Testing](#6-verifikasi--testing)
|
||||||
|
7. [Docker Deployment](#7-docker-deployment)
|
||||||
|
8. [Troubleshooting](#8-troubleshooting)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Fitur
|
||||||
|
|
||||||
- **Framework:** Axum (async Rust web framework)
|
- **Framework:** Axum (async Rust web framework)
|
||||||
- **Runtime Inferensi:** ONNX Runtime untuk kompatibilitas lintas platform
|
- **Runtime Inferensi:** ONNX Runtime untuk kompatibilitas lintas platform
|
||||||
@@ -10,15 +27,13 @@ Layanan inferensi machine learning berbasis Rust dengan Axum web framework dan O
|
|||||||
- **Endpoint:** Health check, metadata, dan prediksi gambar
|
- **Endpoint:** Health check, metadata, dan prediksi gambar
|
||||||
- **Multipart Upload:** Dukungan upload gambar langsung via HTTP POST
|
- **Multipart Upload:** Dukungan upload gambar langsung via HTTP POST
|
||||||
|
|
||||||
## Prasyarat
|
---
|
||||||
|
|
||||||
|
## 2. Prasyarat & Instalasi
|
||||||
|
|
||||||
- Rust 1.70+ dan Cargo
|
- Rust 1.70+ dan Cargo
|
||||||
- Model ONNX di `../../Machine_Learning/model/model.onnx` (atau path custom via `MODEL_PATH`)
|
- Model ONNX di `../../Machine_Learning/model/model.onnx` (atau path custom via `MODEL_PATH`)
|
||||||
|
|
||||||
## Instalasi & Setup
|
|
||||||
|
|
||||||
### Instalasi Dependensi
|
|
||||||
|
|
||||||
Dependensi Rust sudah terdaftar di `Cargo.toml`. Cargo akan mengunduh dan mengkompilasi otomatis saat pertama kali build.
|
Dependensi Rust sudah terdaftar di `Cargo.toml`. Cargo akan mengunduh dan mengkompilasi otomatis saat pertama kali build.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -27,75 +42,58 @@ cargo build
|
|||||||
|
|
||||||
Output build lokal berada di `target/` dan direktori tersebut diabaikan oleh Git.
|
Output build lokal berada di `target/` dan direktori tersebut diabaikan oleh Git.
|
||||||
|
|
||||||
## Menjalankan Service Lokal
|
---
|
||||||
|
|
||||||
|
## 3. Menjalankan Service
|
||||||
|
|
||||||
Semua perintah di bawah dijalankan dari direktori `apps/ml-service`.
|
Semua perintah di bawah dijalankan dari direktori `apps/ml-service`.
|
||||||
|
|
||||||
### Opsi 1: Default (Port 8000, Model dari Machine_Learning/)
|
### Opsi 1: Default (Port 8000)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd apps/ml-service
|
|
||||||
cargo run
|
cargo run
|
||||||
```
|
```
|
||||||
|
|
||||||
Service akan mencari model di path default dan mendengarkan di `http://localhost:8000`:
|
Service akan mencari model di path default:
|
||||||
```
|
```
|
||||||
../../Machine_Learning/model/model.onnx
|
../../Machine_Learning/model/model.onnx
|
||||||
```
|
```
|
||||||
|
|
||||||
### Opsi 2: Local Development dengan .env.example (Port 8001)
|
### Opsi 2: Local Development dengan .env.example (Port 8001)
|
||||||
|
|
||||||
Untuk development lokal dengan port 8001 (sesuai `.env.example`):
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd apps/ml-service
|
|
||||||
source .env.example
|
source .env.example
|
||||||
cargo run
|
cargo run
|
||||||
```
|
```
|
||||||
|
|
||||||
Service akan mendengarkan di `http://localhost:8001` karena `ML_SERVICE_PORT=8001` di `.env.example`.
|
### Opsi 3: Custom Model Path & Port
|
||||||
|
|
||||||
### Opsi 3: Custom Model Path
|
|
||||||
|
|
||||||
Jika model berada di lokasi lain, gunakan environment variable `MODEL_PATH`:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd apps/ml-service
|
|
||||||
MODEL_PATH=/path/to/model.onnx cargo run
|
|
||||||
```
|
|
||||||
|
|
||||||
Atau kombinasikan dengan port custom:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd apps/ml-service
|
|
||||||
ML_SERVICE_PORT=9000 MODEL_PATH=/path/to/model.onnx cargo run
|
ML_SERVICE_PORT=9000 MODEL_PATH=/path/to/model.onnx cargo run
|
||||||
```
|
```
|
||||||
|
|
||||||
## Environment Variables
|
---
|
||||||
|
|
||||||
|
## 4. Environment Variables
|
||||||
|
|
||||||
| Variable | Default | Keterangan |
|
| Variable | Default | Keterangan |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `ML_SERVICE_HOST` | `0.0.0.0` | Bind address |
|
| `ML_SERVICE_HOST` | `0.0.0.0` | Bind address |
|
||||||
| `ML_SERVICE_PORT` | `8000` | Bind port (override untuk local dev dengan `.env.example`) |
|
| `ML_SERVICE_PORT` | `8000` | Bind port |
|
||||||
| `MODEL_PATH` | `../../Machine_Learning/model/model.onnx` | Path ke file model ONNX |
|
| `MODEL_PATH` | `../../Machine_Learning/model/model.onnx` | Path ke file model ONNX |
|
||||||
| `MODEL_INPUT_SIZE` | `224` | Ukuran input gambar (224x224 untuk EfficientNetV2B0) |
|
| `MODEL_INPUT_SIZE` | `224` | Ukuran input gambar (224×224 untuk EfficientNetV2B0) |
|
||||||
| `RUST_LOG` | `info` | Level logging (debug, info, warn, error) |
|
| `RUST_LOG` | `info` | Level logging (debug, info, warn, error) |
|
||||||
|
|
||||||
## Endpoint API
|
---
|
||||||
|
|
||||||
### 1. Health Check
|
## 5. Endpoint API
|
||||||
|
|
||||||
|
### Health Check
|
||||||
|
|
||||||
**Default (port 8000):**
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8000/health
|
curl http://localhost:8000/health
|
||||||
```
|
```
|
||||||
|
|
||||||
**Local dev dengan .env.example (port 8001):**
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8001/health
|
|
||||||
```
|
|
||||||
|
|
||||||
**Response:**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
@@ -103,19 +101,12 @@ curl http://localhost:8001/health
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Metadata
|
### Metadata
|
||||||
|
|
||||||
**Default (port 8000):**
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8000/metadata
|
curl http://localhost:8000/metadata
|
||||||
```
|
```
|
||||||
|
|
||||||
**Local dev dengan .env.example (port 8001):**
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8001/metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
**Response:**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"service_name": "zeavis-ml-service",
|
"service_name": "zeavis-ml-service",
|
||||||
@@ -123,32 +114,19 @@ curl http://localhost:8001/metadata
|
|||||||
"model_path": "../../Machine_Learning/model/model.onnx",
|
"model_path": "../../Machine_Learning/model/model.onnx",
|
||||||
"model_loaded": true,
|
"model_loaded": true,
|
||||||
"input_size": 224,
|
"input_size": 224,
|
||||||
"labels": [
|
"labels": ["Bercak Daun", "Daun Sehat", "Karat Daun", "Hawar Daun"]
|
||||||
"Bercak Daun",
|
|
||||||
"Daun Sehat",
|
|
||||||
"Karat Daun",
|
|
||||||
"Hawar Daun"
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Prediksi
|
### Prediksi
|
||||||
|
|
||||||
Upload gambar daun jagung untuk klasifikasi:
|
Upload gambar daun jagung untuk klasifikasi:
|
||||||
|
|
||||||
**Default (port 8000):**
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8000/predict \
|
curl -X POST http://localhost:8000/predict \
|
||||||
-F "file=@/path/to/corn-leaf.jpg"
|
-F "file=@/path/to/corn-leaf.jpg"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Local dev dengan .env.example (port 8001):**
|
|
||||||
```bash
|
|
||||||
curl -X POST http://localhost:8001/predict \
|
|
||||||
-F "file=@/path/to/corn-leaf.jpg"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Response:**
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"label": "Daun Sehat",
|
"label": "Daun Sehat",
|
||||||
@@ -162,120 +140,45 @@ curl -X POST http://localhost:8001/predict \
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Verifikasi & Testing
|
---
|
||||||
|
|
||||||
|
## 6. Verifikasi & Testing
|
||||||
|
|
||||||
### Build Produksi
|
### Build Produksi
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo build --release
|
cargo build --release
|
||||||
|
# Binary di target/release/zeavis-ml-service
|
||||||
```
|
```
|
||||||
|
|
||||||
Output binary akan tersedia di `target/release/zeavis-ml-service`.
|
|
||||||
|
|
||||||
### Menjalankan Tests
|
### Menjalankan Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo test
|
cargo test
|
||||||
```
|
```
|
||||||
|
|
||||||
Tests mencakup validasi loading model, preprocessing gambar, dan output prediksi.
|
### Verifikasi Manual (default port 8000)
|
||||||
|
|
||||||
### Verifikasi Manual
|
|
||||||
|
|
||||||
#### Dengan default port 8000:
|
|
||||||
|
|
||||||
1. Jalankan service:
|
|
||||||
```bash
|
|
||||||
cargo run
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Di terminal lain, test health endpoint:
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/health
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Test metadata:
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Test prediksi dengan gambar sample:
|
|
||||||
```bash
|
|
||||||
curl -X POST http://localhost:8000/predict \
|
|
||||||
-F "file=@../../Machine_Learning/dataset/Daun\ Sehat/sample.jpg"
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Dengan local dev port 8001 (.env.example):
|
|
||||||
|
|
||||||
1. Jalankan service dengan .env.example:
|
|
||||||
```bash
|
|
||||||
source .env.example
|
|
||||||
cargo run
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Di terminal lain, test health endpoint:
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8001/health
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Test metadata:
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8001/metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Test prediksi dengan gambar sample:
|
|
||||||
```bash
|
|
||||||
curl -X POST http://localhost:8001/predict \
|
|
||||||
-F "file=@../../Machine_Learning/dataset/Daun\ Sehat/sample.jpg"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Model tidak ditemukan
|
|
||||||
|
|
||||||
**Error:** `Failed to load model: No such file or directory`
|
|
||||||
|
|
||||||
**Solusi:** Pastikan file model tersedia di path yang benar:
|
|
||||||
```bash
|
|
||||||
ls -la ../../Machine_Learning/model/model.onnx
|
|
||||||
```
|
|
||||||
|
|
||||||
Atau set path custom:
|
|
||||||
```bash
|
|
||||||
MODEL_PATH=/absolute/path/to/model.onnx cargo run
|
|
||||||
```
|
|
||||||
|
|
||||||
### Port sudah digunakan
|
|
||||||
|
|
||||||
**Error:** `Address already in use`
|
|
||||||
|
|
||||||
**Solusi:** Service menggunakan port 8000 secara default. Jika port sudah digunakan, ubah dengan environment variable:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ML_SERVICE_PORT=9000 cargo run
|
# 1. Start service
|
||||||
|
cargo run
|
||||||
|
|
||||||
|
# 2. Health check
|
||||||
|
curl http://localhost:8000/health
|
||||||
|
|
||||||
|
# 3. Metadata
|
||||||
|
curl http://localhost:8000/metadata
|
||||||
|
|
||||||
|
# 4. Prediksi
|
||||||
|
curl -X POST http://localhost:8000/predict \
|
||||||
|
-F "file=@../../Machine_Learning/dataset/Daun\ Sehat/sample.jpg"
|
||||||
```
|
```
|
||||||
|
|
||||||
Atau jika menggunakan `.env.example` (port 8001), pastikan tidak ada service lain di port tersebut:
|
---
|
||||||
|
|
||||||
```bash
|
## 7. Docker Deployment
|
||||||
lsof -i :8001
|
|
||||||
```
|
|
||||||
|
|
||||||
### ONNX Runtime tidak kompatibel
|
Service dapat di-deploy via Docker. Build dari root repository karena Dockerfile menyalin source service dan artifact ONNX dari beberapa direktori repo.
|
||||||
|
|
||||||
**Error:** `ONNX Runtime initialization failed`
|
|
||||||
|
|
||||||
**Solusi:** Pastikan ONNX Runtime binary kompatibel dengan sistem operasi. Cargo akan mengunduh binary yang sesuai otomatis. Jika masalah persisten, coba rebuild:
|
|
||||||
```bash
|
|
||||||
cargo clean
|
|
||||||
cargo build
|
|
||||||
```
|
|
||||||
|
|
||||||
## Deployment
|
|
||||||
|
|
||||||
### Docker
|
|
||||||
|
|
||||||
Service dapat di-deploy via Docker. Jalankan build dari root repository karena Dockerfile menyalin source service dan artifact ONNX dari beberapa direktori repo.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build -f apps/ml-service/Dockerfile -t zeavis-ml-service .
|
docker build -f apps/ml-service/Dockerfile -t zeavis-ml-service .
|
||||||
@@ -284,11 +187,42 @@ docker run -p 8000:8000 zeavis-ml-service
|
|||||||
|
|
||||||
Pastikan `Machine_Learning/model/model.onnx` sudah dibuat sebelum build image.
|
Pastikan `Machine_Learning/model/model.onnx` sudah dibuat sebelum build image.
|
||||||
|
|
||||||
### Docker Compose
|
---
|
||||||
|
|
||||||
Lihat `docker-compose.yml` di root repository untuk deployment lengkap dengan web, API, dan ML service.
|
## 8. Troubleshooting
|
||||||
|
|
||||||
## Dokumentasi Terkait
|
### Model tidak ditemukan
|
||||||
|
|
||||||
- [`Machine_Learning/README.md`](../../Machine_Learning/README.md) — Panduan training dan ekspor model ONNX
|
**Error:** `Failed to load model: No such file or directory`
|
||||||
- [`README.md`](../../README.md) — Dokumentasi proyek utama
|
|
||||||
|
**Solusi:**
|
||||||
|
```bash
|
||||||
|
ls -la ../../Machine_Learning/model/model.onnx
|
||||||
|
# Atau set path custom:
|
||||||
|
MODEL_PATH=/absolute/path/to/model.onnx cargo run
|
||||||
|
```
|
||||||
|
|
||||||
|
### Port sudah digunakan
|
||||||
|
|
||||||
|
**Error:** `Address already in use`
|
||||||
|
|
||||||
|
**Solusi:**
|
||||||
|
```bash
|
||||||
|
ML_SERVICE_PORT=9000 cargo run
|
||||||
|
# Cek port yang digunakan:
|
||||||
|
lsof -i :8000
|
||||||
|
```
|
||||||
|
|
||||||
|
### ONNX Runtime tidak kompatibel
|
||||||
|
|
||||||
|
**Error:** `ONNX Runtime initialization failed`
|
||||||
|
|
||||||
|
**Solusi:** Pastikan binary ONNX Runtime kompatibel dengan sistem operasi. Jika masalah persisten:
|
||||||
|
```bash
|
||||||
|
cargo clean
|
||||||
|
cargo build
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
← [Kembali ke README utama](../../README.md) • [Pipeline ML →](../../Machine_Learning/README.md) • [Infra →](../../infra/README.md)
|
||||||
|
|||||||
+80
-33
@@ -1,6 +1,25 @@
|
|||||||
# Infra — ZeaVis Edu Multi-VPS Deployment
|
# Infrastruktur — ZeaVis Edu
|
||||||
|
|
||||||
## Arsitektur
|
> Arsitektur multi-VPS untuk deployment produksi ZeaVis Edu dengan Tailscale mesh VPN dan observabilitas penuh.
|
||||||
|
|
||||||
|
← [Kembali ke README utama](../README.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Daftar Isi
|
||||||
|
|
||||||
|
1. [Arsitektur](#1-arsitektur)
|
||||||
|
2. [Prasyarat GitHub Secrets](#2-prasyarat-github-secrets)
|
||||||
|
3. [Setup VPS](#3-setup-vps)
|
||||||
|
4. [Port yang Dibuka](#4-port-yang-dibuka)
|
||||||
|
5. [Metrics Flow](#5-metrics-flow)
|
||||||
|
6. [Perintah Penting](#6-perintah-penting)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Arsitektur
|
||||||
|
|
||||||
|
ZeaVis Edu berjalan di **dua VPS terpisah** yang terhubung melalui **Tailscale** mesh VPN:
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐
|
||||||
@@ -44,32 +63,41 @@
|
|||||||
└─────────────────────────────────────────────┘ └──────────────────────────────────────────────┘
|
└─────────────────────────────────────────────┘ └──────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
## Prerequisites
|
| VPS | Hostname | OS | Peran |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **App VPS** | `imrnes` | Arch Linux | Web (:80), API (:3000), ML Service (:8000) |
|
||||||
|
| **Telemetry VPS** | `orange` | Ubuntu | Prometheus, ClickHouse, Telemetry UI |
|
||||||
|
|
||||||
### GitHub Secrets (untuk CI/CD)
|
---
|
||||||
|
|
||||||
**App VPS deploy (`.github/workflows/deploy.yml`):**
|
## 2. Prasyarat GitHub Secrets
|
||||||
| Secret | Value |
|
|
||||||
|--------|-------|
|
### App VPS — `.github/workflows/deploy.yml`
|
||||||
|
|
||||||
|
| Secret | Keterangan |
|
||||||
|
|---|---|
|
||||||
| `VPS_HOST` | `100.108.1.124` (imrnes) |
|
| `VPS_HOST` | `100.108.1.124` (imrnes) |
|
||||||
| `VPS_USER` | `mytheclipse` |
|
| `VPS_USER` | `mytheclipse` |
|
||||||
| `VPS_SSH_KEY` | Private SSH key for imrnes |
|
| `VPS_SSH_KEY` | Private SSH key untuk imrnes |
|
||||||
| `VPS_PORT` | `22` |
|
| `VPS_PORT` | `22` |
|
||||||
| `DATABASE_URL` | PostgreSQL connection string |
|
| `DATABASE_URL` | PostgreSQL connection string |
|
||||||
| `SESSION_SECRET` | Random session secret |
|
| `SESSION_SECRET` | Random session secret |
|
||||||
|
|
||||||
**Telemetry VPS deploy (`.github/workflows/telemetry-ci-cd.yml`):**
|
### Telemetry VPS — `.github/workflows/telemetry-ci-cd.yml`
|
||||||
| Secret | Value |
|
|
||||||
|--------|-------|
|
| Secret | Keterangan |
|
||||||
|
|---|---|
|
||||||
| `TELEMETRY_VPS_HOST` | `100.96.248.86` (orange) |
|
| `TELEMETRY_VPS_HOST` | `100.96.248.86` (orange) |
|
||||||
| `TELEMETRY_VPS_USER` | SSH username for orange |
|
| `TELEMETRY_VPS_USER` | SSH username |
|
||||||
| `TELEMETRY_VPS_SSH_KEY` | Private SSH key for orange |
|
| `TELEMETRY_VPS_SSH_KEY` | Private SSH key |
|
||||||
| `TELEMETRY_VPS_PORT` | `22` |
|
| `TELEMETRY_VPS_PORT` | `22` |
|
||||||
| `GHCR_PAT` | GitHub PAT with `write:packages` + `read:packages` |
|
| `GHCR_PAT` | GitHub PAT dengan `write:packages` + `read:packages` |
|
||||||
|
|
||||||
### VPS Setup
|
---
|
||||||
|
|
||||||
#### 1. App VPS (imrnes — 100.108.1.124)
|
## 3. Setup VPS
|
||||||
|
|
||||||
|
### App VPS (imrnes — 100.108.1.124)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Create Docker network
|
# Create Docker network
|
||||||
@@ -79,33 +107,35 @@ docker network create telemetry-net
|
|||||||
# ZeaVis Edu apps deploy automatically via GitHub Actions
|
# ZeaVis Edu apps deploy automatically via GitHub Actions
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 2. Telemetry VPS (orange — 100.96.248.86)
|
### Telemetry VPS (orange — 100.96.248.86)
|
||||||
|
|
||||||
Deploy via GitHub Actions workflow `.github/workflows/telemetry-ci-cd.yml`.
|
Deploy via GitHub Actions atau manual:
|
||||||
|
|
||||||
Atau manual:
|
|
||||||
```bash
|
```bash
|
||||||
ssh mytheclipse@100.96.248.86
|
ssh mytheclipse@100.96.248.86
|
||||||
mkdir -p /opt/telemetry
|
mkdir -p /opt/telemetry
|
||||||
# ... sync files from telemetry/ directory ...
|
|
||||||
cd /opt/telemetry
|
cd /opt/telemetry
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
bash clickhouse/init.sh
|
bash clickhouse/init.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
## Port yang dibuka
|
---
|
||||||
|
|
||||||
|
## 4. Port yang Dibuka
|
||||||
|
|
||||||
### App VPS (imrnes)
|
### App VPS (imrnes)
|
||||||
|
|
||||||
| Port | Service | Akses |
|
| Port | Service | Akses |
|
||||||
|------|---------|-------|
|
|---|---|---|
|
||||||
| 80/443 | Web (via Traefik/Coolify) | Public |
|
| 80/443 | Web (via Traefik/Coolify) | Public |
|
||||||
| 3000 | API metrics | Tailscale-only |
|
| 3000 | API metrics | Tailscale-only |
|
||||||
| 8000 | ML service metrics | Tailscale-only |
|
| 8000 | ML service metrics | Tailscale-only |
|
||||||
| 9100 | Node Exporter | Tailscale-only |
|
| 9100 | Node Exporter | Tailscale-only |
|
||||||
|
|
||||||
### Telemetry VPS (orange)
|
### Telemetry VPS (orange)
|
||||||
|
|
||||||
| Port | Service | Akses |
|
| Port | Service | Akses |
|
||||||
|------|---------|-------|
|
|---|---|---|
|
||||||
| 80/443 | Telemetry UI (via Coolify Traefik) | Public |
|
| 80/443 | Telemetry UI (via Coolify Traefik) | Public |
|
||||||
| 8181 | Telemetry UI (direct) | Tailscale-only |
|
| 8181 | Telemetry UI (direct) | Tailscale-only |
|
||||||
| 9090 | Prometheus | Tailscale-only |
|
| 9090 | Prometheus | Tailscale-only |
|
||||||
@@ -114,27 +144,44 @@ bash clickhouse/init.sh
|
|||||||
| 8123 | ClickHouse HTTP | Tailscale-only |
|
| 8123 | ClickHouse HTTP | Tailscale-only |
|
||||||
| 9000 | ClickHouse Native | Tailscale-only |
|
| 9000 | ClickHouse Native | Tailscale-only |
|
||||||
|
|
||||||
## Metrics Flow
|
---
|
||||||
|
|
||||||
1. **App services** expose `/metrics` pada port masing-masing
|
## 5. Metrics Flow
|
||||||
|
|
||||||
|
1. **App services** mengekspos `GET /metrics` di port masing-masing
|
||||||
2. **Prometheus** di orange VPS scrape via Tailscale IP (`100.108.1.124:PORT`)
|
2. **Prometheus** di orange VPS scrape via Tailscale IP (`100.108.1.124:PORT`)
|
||||||
3. **Prometheus** forward ke **Metric Ingester** via `remote_write`
|
3. Prometheus forward ke **Metric Ingester** via `remote_write`
|
||||||
4. **Metric Ingester** enrich → filter → forward ke **Vector**
|
4. Metric Ingester enrich → filter → forward ke **Vector**
|
||||||
5. **Vector** buffer → write ke **ClickHouse**
|
5. Vector buffer → write ke **ClickHouse**
|
||||||
6. **Telemetry UI** query via **Query Proxy** → **ClickHouse**
|
6. **Telemetry UI** query via **Query Proxy** → **ClickHouse**
|
||||||
|
|
||||||
## Useful Commands
|
```
|
||||||
|
App Services (/metrics)
|
||||||
|
│
|
||||||
|
▼ (scrape via Tailscale)
|
||||||
|
Prometheus ──(remote_write)──► Metric Ingester ──► Vector ──► ClickHouse
|
||||||
|
│
|
||||||
|
Query Proxy ◄── Telemetry UI
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Perintah Penting
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Telemetry stack status
|
# Status telemetry stack
|
||||||
make telemetry-status
|
make telemetry-status
|
||||||
|
|
||||||
# View telemetry logs
|
# Lihat log service tertentu
|
||||||
make telemetry-logs s=prometheus
|
make telemetry-logs s=prometheus
|
||||||
|
|
||||||
# Send test metric
|
# Kirim test metric
|
||||||
make telemetry-test-metric
|
make telemetry-test-metric
|
||||||
|
|
||||||
# Restart a service
|
# Restart service
|
||||||
make telemetry-restart s=vector
|
make telemetry-restart s=vector
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
← [Kembali ke README utama](../README.md) • [ML Service →](../apps/ml-service/README.md) • [Pipeline ML →](../Machine_Learning/README.md)
|
||||||
|
|||||||
Reference in New Issue
Block a user