From 6933e5512f0a660972e9b43afe19489d3205f87c Mon Sep 17 00:00:00 2001 From: MythEclipse Date: Sun, 7 Jun 2026 21:15:32 +0700 Subject: [PATCH] docs(telemetry): separate production cross-VPS and local dev architecture docs --- CLAUDE.md | 4 ++++ METRICS.md | 20 ++++++++++++------ Makefile | 14 +++++++------ README.md | 61 ++++++++++++++++++++++++++++++++++-------------------- 4 files changed, 65 insertions(+), 34 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4adcacf..196231f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,8 +151,12 @@ The repository includes a full Prometheus → ClickHouse metric pipeline as a gi All three share the `zeavis_` metric prefix and are scraped by the Telemetry Prometheus instance via `file_sd_configs` (see `telemetry/prometheus/targets/zeavis-edu.json`). +**IMPORTANT — Production architecture:** ZeaVis Edu apps and the Telemetry stack run on **separate VPS instances** connected via **Tailscale** (mesh VPN). Prometheus scrapes the API and ML service through their **Tailscale IPs** (e.g. `100.x.x.a:3000`), not via Docker hostnames. The target file `telemetry/prometheus/targets/zeavis-edu.json` has `__CHANGE_ME__` placeholders — before deploying, replace with the actual Tailscale IPs of the app VPS. + The telemetry stack is managed from the project root via `make telemetry-*` targets (see `Makefile`). The Docker Compose files in `telemetry/deploy/` define 6 services (Prometheus, Metric Ingester, Vector, ClickHouse, Query Proxy, Telemetry UI). +For **local single-host dev**, Prometheus can reach app services via a shared Docker network (`app-shared-net`). Use `make telemetry-up-local` for this mode — it includes the `docker-compose.telemetry.yml` override. + ## Fullstack application architecture The root TypeScript workspace is a Bun + Moon monorepo: diff --git a/METRICS.md b/METRICS.md index 2e1f21d..af8ba31 100644 --- a/METRICS.md +++ b/METRICS.md @@ -15,8 +15,11 @@ application stack and the payload each service provides. | Prometheus Collector | — | `GET /metrics` (self) | 9090 | > In production all metrics are scraped by the Prometheus collector running in the -> Telemetry stack. See [`telemetry/prometheus/targets/`](./telemetry/prometheus/targets/) -> for the auto‑discovery configuration. +> Telemetry stack on a **separate VPS** connected via **Tailscale**. +> See [`telemetry/prometheus/targets/`](./telemetry/prometheus/targets/) +> for the auto‑discovery configuration. Target files must use **Tailscale IPs** +> (e.g. `100.x.x.a:3000`), not Docker hostnames, because the services are on +> different hosts. > > In production (nginx), the web app proxies `/metrics` to the API service: > see [`apps/web/nginx.conf`](apps/web/nginx.conf). @@ -98,16 +101,21 @@ The Telemetry submodule includes a Prometheus instance that uses ```json [ { - "targets": ["zeavis-api:3000"], - "labels": { "service": "zeavis-api", "component": "backend" } + "targets": ["100.x.x.a:3000"], + "labels": { "service": "zeavis-api", "component": "backend", "env": "production" } }, { - "targets": ["zeavis-ml:8000"], - "labels": { "service": "zeavis-ml", "component": "inference" } + "targets": ["100.x.x.b:8000"], + "labels": { "service": "zeavis-ml", "component": "inference", "env": "production" } } ] ``` +> ⚠️ **Cross-VPS:** Gunakan **IP Tailscale** (bukan Docker hostname) karena +> Prometheus dan ZeaVis Edu berjalan di VPS berbeda. Pastikan port service +> (`:3000`, `:8000`) terekspos di `0.0.0.0` atau diizinkan oleh aturan +> `iptables`/`ufw` untuk interface Tailscale (`tailscale0`/`100.x.x.x/10`). + The Prometheus config (in `telemetry/prometheus/prometheus.yml`) will automatically pick up new files within its 15‑second scrape interval — no restart required. diff --git a/Makefile b/Makefile index 3c6c015..8559fa3 100644 --- a/Makefile +++ b/Makefile @@ -44,21 +44,23 @@ TELEMETRY_COMPOSE := telemetry/deploy/docker-compose.yml TELEMETRY_LOCAL := telemetry/deploy/docker-compose.local.yml TELEMETRY_ZEAVIS := docker-compose.telemetry.yml -# Start all telemetry services +# Start all telemetry services (standalone — cross-VPS production mode) +# Prometheus scrapes ZeaVis Edu via Tailscale IPs, not Docker network. telemetry-up: - @echo ">> Starting Telemetry stack..." + @echo ">> Starting Telemetry stack (standalone)..." CLICKHOUSE_USER=$${CLICKHOUSE_USER:-telemetry} \ CLICKHOUSE_PASSWORD=$${CLICKHOUSE_PASSWORD:-telemetry} \ - docker compose -f $(TELEMETRY_COMPOSE) -f $(TELEMETRY_ZEAVIS) up -d + docker compose -f $(TELEMETRY_COMPOSE) up -d @echo ">> Telemetry stack started. Use 'make telemetry-logs' to view output." -# Start telemetry services with local port overrides (no Tailscale) +# Start telemetry services with ZeaVis Edu network sharing (local single-host dev) +# Prometheus can scrape app services via app-shared-net Docker network. telemetry-up-local: - @echo ">> Starting Telemetry stack (local mode)..." + @echo ">> Starting Telemetry stack (local dev mode)..." CLICKHOUSE_USER=$${CLICKHOUSE_USER:-telemetry} \ CLICKHOUSE_PASSWORD=$${CLICKHOUSE_PASSWORD:-telemetry} \ docker compose -f $(TELEMETRY_COMPOSE) -f $(TELEMETRY_LOCAL) -f $(TELEMETRY_ZEAVIS) up -d - @echo ">> Telemetry stack started in local mode." + @echo ">> Telemetry stack started in local dev mode." # Stop all telemetry services telemetry-down: diff --git a/README.md b/README.md index c134bc0..a50dbf4 100644 --- a/README.md +++ b/README.md @@ -218,33 +218,34 @@ docker compose up -d Proyek ini menyertakan pipeline telemetry metric sebagai git submodule di `telemetry/`. Pipeline mengalirkan metrik dari seluruh service ZeaVis Edu ke ClickHouse untuk analisis dan visualisasi jangka panjang. -### Arsitektur +### Arsitektur (Production) + +Di production, aplikasi dan telemetry berjalan di **VPS terpisah** dan terhubung via **Tailscale** (mesh VPN). Prometheus di VPS telemetry melakukan scrape ke service ZeaVis Edu melalui IP Tailscale masing-masing. ```mermaid flowchart LR - subgraph Apps["ZeaVis Edu"] - W[Web / React] - A[API / Elysia] - M[ML Service / Axum] + subgraph VPS1["VPS — ZeaVis Edu (App)"] + W[Web / React
api-zeavisedu.asepharyana.id] + A[API / Elysia
:3000] + M[ML Service / Axum
:8000] end - subgraph Telemetry["Telemetry Pipeline"] - P[Prometheus] - MI[Metric Ingester] - V[Vector] - CH[ClickHouse] - QP[Query Proxy] - TUI[Telemetry UI] + subgraph VPS2["VPS — Telemetry Stack"] + P[Prometheus
:9090] + MI[Metric Ingester
:9091] + V[Vector
:9001] + CH[ClickHouse
:8123] + QP[Query Proxy
:9092] + TUI[Telemetry UI
:8181] end - W -->|"GET /metrics"| P - A -->|"GET /metrics"| P - M -->|"GET /metrics"| P + P -.->|"scrape via Tailscale IP
100.x.x.a:3000/metrics"| A + P -.->|"scrape via Tailscale IP
100.x.x.a:8000/metrics"| M P -->|remote_write| MI - MI -->|HTTP POST| V - V -->|JSONEachRow| CH - QP -->|SQL| CH - TUI -->|/proxy/query| QP + MI --> V + V --> CH + QP --> CH + TUI --> QP ``` Setiap service ZeaVis Edu mengekspos endpoint `/metrics` dalam format Prometheus text: @@ -255,6 +256,8 @@ Setiap service ZeaVis Edu mengekspos endpoint `/metrics` dalam format Prometheus | API (Elysia) | `GET /metrics` | 3000 | | ML Service (Axum) | `GET /metrics` | 8000 | +Prometheus di VPS telemetry melakukan **scrape langsung** ke API dan ML service melalui IP Tailscale mereka, bukan melalui domain publik. Konfigurasi target ada di `telemetry/prometheus/targets/zeavis-edu.json` — isi dengan IP Tailscale dari service yang dituju. + Lihat [`METRICS.md`](./METRICS.md) untuk daftar lengkap metrik yang diekspos. ### Service Telemetry @@ -268,6 +271,18 @@ Lihat [`METRICS.md`](./METRICS.md) untuk daftar lengkap metrik yang diekspos. | 5 | **Query Proxy** | Read-only SQL proxy, tenant isolation | 9092 | | 6 | **Telemetry UI** | Vue 3 metrics dashboard | 8181 | +### Arsitektur (Local Dev) + +Untuk development lokal di satu mesin, telemetry dan app bisa jalan bareng di satu Docker host. Prometheus bisa scrape service lewat Docker network yang sama. + +```bash +# Setup network +docker network create app-shared-net + +# Build & start telemetry (dengan network sharing) +make telemetry-up-local +``` + ### Menjalankan Telemetry Stack Semua operasi telemetry dijalankan dari **root proyek** melalui Makefile: @@ -317,15 +332,17 @@ make telemetry-up-local Prometheus menggunakan `file_sd_configs` untuk menemukan target secara dinamis. Cukup letakkan file JSON di `telemetry/prometheus/targets/` dan Prometheus akan otomatis mendeteksinya dalam 15 detik — tanpa restart. -File target ZeaVis Edu sudah tersedia di [`telemetry/prometheus/targets/zeavis-edu.json`](telemetry/prometheus/targets/zeavis-edu.json): +File template sudah tersedia di [`telemetry/prometheus/targets/zeavis-edu.json`](telemetry/prometheus/targets/zeavis-edu.json). **Sebelum production, isi `__CHANGE_ME__` dengan IP Tailscale masing-masing service:** ```json [ - { "targets": ["zeavis-api:3000"], "labels": { "service": "zeavis-api", "component": "backend" } }, - { "targets": ["zeavis-ml:8000"], "labels": { "service": "zeavis-ml", "component": "inference" } } + { "targets": ["100.x.x.a:3000"], "labels": { "service": "zeavis-api", "component": "backend", "env": "production" } }, + { "targets": ["100.x.x.a:8000"], "labels": { "service": "zeavis-ml", "component": "inference", "env": "production" } } ] ``` +> **Catatan:** Aplikasi ZeaVis Edu mengekspose port Docker-nya (`:3000`, `:8000`) langsung ke host via `docker-compose.yml`. Pastikan port-port tersebut terbuka di network Tailscale (biasanya iptables Tailscale mengizinkan koneksi ke port localhost). + ### Environment Variables Telemetry | Variable | Default | Deskripsi |