docs(telemetry): separate production cross-VPS and local dev architecture docs

2026-06-07 21:15:32 +07:00
parent 7e0e052200
commit 6933e5512f
4 changed files with 65 additions and 34 deletions
@@ -151,8 +151,12 @@ The repository includes a full Prometheus → ClickHouse metric pipeline as a gi

 All three share the `zeavis_` metric prefix and are scraped by the Telemetry Prometheus instance via `file_sd_configs` (see `telemetry/prometheus/targets/zeavis-edu.json`).

+**IMPORTANT — Production architecture:** ZeaVis Edu apps and the Telemetry stack run on **separate VPS instances** connected via **Tailscale** (mesh VPN). Prometheus scrapes the API and ML service through their **Tailscale IPs** (e.g. `100.x.x.a:3000`), not via Docker hostnames. The target file `telemetry/prometheus/targets/zeavis-edu.json` has `__CHANGE_ME__` placeholders — before deploying, replace with the actual Tailscale IPs of the app VPS.
+
 The telemetry stack is managed from the project root via `make telemetry-*` targets (see `Makefile`). The Docker Compose files in `telemetry/deploy/` define 6 services (Prometheus, Metric Ingester, Vector, ClickHouse, Query Proxy, Telemetry UI).

+For **local single-host dev**, Prometheus can reach app services via a shared Docker network (`app-shared-net`). Use `make telemetry-up-local` for this mode — it includes the `docker-compose.telemetry.yml` override.
+
 ## Fullstack application architecture

 The root TypeScript workspace is a Bun + Moon monorepo:
@@ -15,8 +15,11 @@ application stack and the payload each service provides.
 | Prometheus Collector  | —                                 | `GET /metrics` (self)      | 9090         |

 > In production all metrics are scraped by the Prometheus collector running in the
-> Telemetry stack. See [`telemetry/prometheus/targets/`](./telemetry/prometheus/targets/)
-> for the auto‑discovery configuration.
+> Telemetry stack on a **separate VPS** connected via **Tailscale**.
+> See [`telemetry/prometheus/targets/`](./telemetry/prometheus/targets/)
+> for the auto‑discovery configuration. Target files must use **Tailscale IPs**
+> (e.g. `100.x.x.a:3000`), not Docker hostnames, because the services are on
+> different hosts.
 >
 > In production (nginx), the web app proxies `/metrics` to the API service:
 > see [`apps/web/nginx.conf`](apps/web/nginx.conf).
@@ -98,16 +101,21 @@ The Telemetry submodule includes a Prometheus instance that uses
 ```json
 [
  {
-    "targets": ["zeavis-api:3000"],
-    "labels": { "service": "zeavis-api", "component": "backend" }
+    "targets": ["100.x.x.a:3000"],
+    "labels": { "service": "zeavis-api", "component": "backend", "env": "production" }
  },
  {
-    "targets": ["zeavis-ml:8000"],
-    "labels": { "service": "zeavis-ml", "component": "inference" }
+    "targets": ["100.x.x.b:8000"],
+    "labels": { "service": "zeavis-ml", "component": "inference", "env": "production" }
  }
 ]
 ```

+> ⚠️ **Cross-VPS:** Gunakan **IP Tailscale** (bukan Docker hostname) karena
+> Prometheus dan ZeaVis Edu berjalan di VPS berbeda. Pastikan port service
+> (`:3000`, `:8000`) terekspos di `0.0.0.0` atau diizinkan oleh aturan
+> `iptables`/`ufw` untuk interface Tailscale (`tailscale0`/`100.x.x.x/10`).
+
 The Prometheus config (in `telemetry/prometheus/prometheus.yml`) will
 automatically pick up new files within its 15‑second scrape interval —
 no restart required.
@@ -44,21 +44,23 @@ TELEMETRY_COMPOSE := telemetry/deploy/docker-compose.yml
 TELEMETRY_LOCAL   := telemetry/deploy/docker-compose.local.yml
 TELEMETRY_ZEAVIS  := docker-compose.telemetry.yml

-# Start all telemetry services
+# Start all telemetry services (standalone — cross-VPS production mode)
+# Prometheus scrapes ZeaVis Edu via Tailscale IPs, not Docker network.
 telemetry-up:
-	@echo ">> Starting Telemetry stack..."
+	@echo ">> Starting Telemetry stack (standalone)..."
 	CLICKHOUSE_USER=$${CLICKHOUSE_USER:-telemetry} \
 	CLICKHOUSE_PASSWORD=$${CLICKHOUSE_PASSWORD:-telemetry} \
-	docker compose -f $(TELEMETRY_COMPOSE) -f $(TELEMETRY_ZEAVIS) up -d
+	docker compose -f $(TELEMETRY_COMPOSE) up -d
 	@echo ">> Telemetry stack started. Use 'make telemetry-logs' to view output."

-# Start telemetry services with local port overrides (no Tailscale)
+# Start telemetry services with ZeaVis Edu network sharing (local single-host dev)
+# Prometheus can scrape app services via app-shared-net Docker network.
 telemetry-up-local:
-	@echo ">> Starting Telemetry stack (local mode)..."
+	@echo ">> Starting Telemetry stack (local dev mode)..."
 	CLICKHOUSE_USER=$${CLICKHOUSE_USER:-telemetry} \
 	CLICKHOUSE_PASSWORD=$${CLICKHOUSE_PASSWORD:-telemetry} \
 	docker compose -f $(TELEMETRY_COMPOSE) -f $(TELEMETRY_LOCAL) -f $(TELEMETRY_ZEAVIS) up -d
-	@echo ">> Telemetry stack started in local mode."
+	@echo ">> Telemetry stack started in local dev mode."

 # Stop all telemetry services
 telemetry-down:
@@ -218,33 +218,34 @@ docker compose up -d

 Proyek ini menyertakan pipeline telemetry metric sebagai git submodule di `telemetry/`. Pipeline mengalirkan metrik dari seluruh service ZeaVis Edu ke ClickHouse untuk analisis dan visualisasi jangka panjang.

-### Arsitektur
+### Arsitektur (Production)
+
+Di production, aplikasi dan telemetry berjalan di **VPS terpisah** dan terhubung via **Tailscale** (mesh VPN). Prometheus di VPS telemetry melakukan scrape ke service ZeaVis Edu melalui IP Tailscale masing-masing.

 ```mermaid
 flowchart LR
-    subgraph Apps["ZeaVis Edu"]
-        W[Web / React]
-        A[API / Elysia]
-        M[ML Service / Axum]
+    subgraph VPS1["VPS — ZeaVis Edu (App)"]
+        W[Web / React<br/>api-zeavisedu.asepharyana.id]
+        A[API / Elysia<br/>:3000]
+        M[ML Service / Axum<br/>:8000]
    end

-    subgraph Telemetry["Telemetry Pipeline"]
-        P[Prometheus]
-        MI[Metric Ingester]
-        V[Vector]
-        CH[ClickHouse]
-        QP[Query Proxy]
-        TUI[Telemetry UI]
+    subgraph VPS2["VPS — Telemetry Stack"]
+        P[Prometheus<br/>:9090]
+        MI[Metric Ingester<br/>:9091]
+        V[Vector<br/>:9001]
+        CH[ClickHouse<br/>:8123]
+        QP[Query Proxy<br/>:9092]
+        TUI[Telemetry UI<br/>:8181]
    end

-    W -->|"GET /metrics"| P
-    A -->|"GET /metrics"| P
-    M -->|"GET /metrics"| P
+    P -.->|"scrape via Tailscale IP<br/>100.x.x.a:3000/metrics"| A
+    P -.->|"scrape via Tailscale IP<br/>100.x.x.a:8000/metrics"| M
    P -->|remote_write| MI
-    MI -->|HTTP POST| V
-    V -->|JSONEachRow| CH
-    QP -->|SQL| CH
-    TUI -->|/proxy/query| QP
+    MI --> V
+    V --> CH
+    QP --> CH
+    TUI --> QP
 ```

 Setiap service ZeaVis Edu mengekspos endpoint `/metrics` dalam format Prometheus text:
@@ -255,6 +256,8 @@ Setiap service ZeaVis Edu mengekspos endpoint `/metrics` dalam format Prometheus
 | API (Elysia)          | `GET /metrics`     | 3000         |
 | ML Service (Axum)     | `GET /metrics`     | 8000         |

+Prometheus di VPS telemetry melakukan **scrape langsung** ke API dan ML service melalui IP Tailscale mereka, bukan melalui domain publik. Konfigurasi target ada di `telemetry/prometheus/targets/zeavis-edu.json` — isi dengan IP Tailscale dari service yang dituju.
+
 Lihat [`METRICS.md`](./METRICS.md) untuk daftar lengkap metrik yang diekspos.

 ### Service Telemetry
@@ -268,6 +271,18 @@ Lihat [`METRICS.md`](./METRICS.md) untuk daftar lengkap metrik yang diekspos.
 | 5 | **Query Proxy** | Read-only SQL proxy, tenant isolation | 9092 |
 | 6 | **Telemetry UI** | Vue 3 metrics dashboard | 8181 |

+### Arsitektur (Local Dev)
+
+Untuk development lokal di satu mesin, telemetry dan app bisa jalan bareng di satu Docker host. Prometheus bisa scrape service lewat Docker network yang sama.
+
+```bash
+# Setup network
+docker network create app-shared-net
+
+# Build & start telemetry (dengan network sharing)
+make telemetry-up-local
+```
+
 ### Menjalankan Telemetry Stack

 Semua operasi telemetry dijalankan dari **root proyek** melalui Makefile:
@@ -317,15 +332,17 @@ make telemetry-up-local

 Prometheus menggunakan `file_sd_configs` untuk menemukan target secara dinamis. Cukup letakkan file JSON di `telemetry/prometheus/targets/` dan Prometheus akan otomatis mendeteksinya dalam 15 detik — tanpa restart.

-File target ZeaVis Edu sudah tersedia di [`telemetry/prometheus/targets/zeavis-edu.json`](telemetry/prometheus/targets/zeavis-edu.json):
+File template sudah tersedia di [`telemetry/prometheus/targets/zeavis-edu.json`](telemetry/prometheus/targets/zeavis-edu.json). **Sebelum production, isi `__CHANGE_ME__` dengan IP Tailscale masing-masing service:**

 ```json
 [
-  { "targets": ["zeavis-api:3000"], "labels": { "service": "zeavis-api", "component": "backend" } },
-  { "targets": ["zeavis-ml:8000"],  "labels": { "service": "zeavis-ml", "component": "inference" } }
+  { "targets": ["100.x.x.a:3000"], "labels": { "service": "zeavis-api", "component": "backend", "env": "production" } },
+  { "targets": ["100.x.x.a:8000"],  "labels": { "service": "zeavis-ml", "component": "inference", "env": "production" } }
 ]
 ```

+> **Catatan:** Aplikasi ZeaVis Edu mengekspose port Docker-nya (`:3000`, `:8000`) langsung ke host via `docker-compose.yml`. Pastikan port-port tersebut terbuka di network Tailscale (biasanya iptables Tailscale mengizinkan koneksi ke port localhost).
+
 ### Environment Variables Telemetry

 | Variable | Default | Deskripsi |