From 5025d103e9482b695a7561e5dd272e63f8e00464 Mon Sep 17 00:00:00 2001 From: asepharyana Date: Wed, 22 Jul 2026 15:16:36 +0700 Subject: [PATCH] chore: update scraper submodule to 7640e66 and add deployment documentation --- CLAUDE.md | 118 +++++++++++++++++++ docs/DEPLOYMENT.md | 67 +++++++++++ docs/adr/0002-env-file-via-github-secret.md | 122 ++++++++++++++++++++ 3 files changed, 307 insertions(+) create mode 100644 CLAUDE.md create mode 100644 docs/adr/0002-env-file-via-github-secret.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..a6f0ca6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,118 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository Overview + +Asepharyana Hub is a **hub monorepo** for Asep Haryana Saputra's portfolio ecosystem. Application services live in separate repos imported as Git submodules under `apps/`. Infrastructure (Docker Compose, Traefik, Dapr) lives in `infra/`. + +``` +asepharyana-hub/ +├── apps/ # Git submodules — each app is its own repo +│ └── scraper/ # Rust scraper API (asepharyana-hub-scraper) +├── docs/ # ADRs, deployment guide, new-app guide +├── infra/ +│ ├── compose/ # One Docker Compose file per service +│ ├── dapr/ # Dapr config + component definitions +│ ├── docker/ # Dockerfiles per service +│ └── traefik/ # Reverse proxy config (static + dynamic) +├── scripts/ # Utility scripts (cleanup, update-deps, git hooks) +└── .github/workflows/ # CI/CD pipelines +``` + +### Submodule Strategy +- Each app in `apps/` is a separate Git repo imported as a submodule. Code changes happen in the submodule repo, not here. +- Submodule pointers are updated by CI/CD (via `repository_dispatch` or manual commit). +- Current submodule: `apps/scraper` → `asepharyana/asepharyana-hub-scraper`. + +### Infrastructure Stack +- **Traefik v3.6** — reverse proxy, TLS termination, middleware chain (rate-limit, headers, buffer, block-sensitive-paths) +- **NATS + JetStream** — message broker with persistent streaming +- **Dapr** — sidecar runtime (pub/sub abstraction, state management, service invocation) +- **Redis (Alpine)** — cache, session store, Dapr state store & pub/sub backend +- **Tailscale** — secure overlay network between VPS nodes (PostgreSQL on `imrnes`, containers on `orangevps`) + +### Networking +- All containers join `app-shared-net` (external Docker bridge network). Service discovery via Docker DNS (container name aliases). +- Traefik handles all external HTTP/S traffic on port 443. +- Cross-VPS traffic (DB, Redis) goes through Tailscale (`100.64.0.0/10`). Container-to-Tailscale connectivity requires a route in the main routing table (managed by `tailscale-routes.service`). + +## Commands + +```bash +make init-submodules # Initialize submodules after clone +make dev # Start shared dev infrastructure (Redis) +make update-submodules # Update all submodules to latest + +bun run check # Biome lint + format + write +bun run ci # Biome CI mode (no writes, exit code on issues) +bun run format # Format only +bun run lint # Lint only + +docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest . # Build image +``` + +### Validate YAML +```bash +python -c "import pathlib, yaml; [yaml.safe_load(open(p)) for p in pathlib.Path('infra').rglob('*.yml')]" +for f in infra/compose/*.yml; do docker compose -f "$f" config >/dev/null && echo "OK $f"; done +``` + +## CI/CD Workflows + +| Workflow | Trigger | Action | +|----------|---------|--------| +| `lint.yml` | PR/push to main touching `*.json`, `*.js`, `biome.json` | `bun run ci` (Biome lint) | +| `docker-build-push.yml` | Push to main touching `apps/**`/`infra/**`, or `repository_dispatch` | Build Docker images per changed service, push to GHCR, update compose manifests | +| `deploy-docker.yml` | After build completes, or push touching `infra/**` | SSH to VPS (orangevps), pull images, restart containers selectively | +| `security.yml` | PR to main + weekly Monday | CodeQL analysis (Rust) | +| `update-submodule.yml` | `repository_dispatch` | Update submodule pointer in hub repo | + +### Deployment Order +1. `shared.yml` (Redis) +2. `nats.yml` (NATS + JetStream) +3. `dapr.yml` (Dapr placement) +4. `traefik.yml` (Reverse proxy) +5. Service compose files (app + Dapr sidecar) + +### Secrets Required for Deploy +`SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`, `VPS_TARGET_DIR`, `ENV_FILE_PRODUCTION` + +## Infrastructure Patterns + +### Compose File Pattern +Each service gets one compose file. Containers join `app-shared-net` with a `container_name` alias for DNS. The network is declared `external: true`. + +### Dapr Sidecar Pattern +Each app gets a companion `daprd` sidecar container. Dapr components (pubsub, statestore) are mounted from `infra/dapr/components/`. The sidecar communicates with NATS for pub/sub and Dapr placement for actor coordination. + +### Traefik Routing +- Routers + services defined in `infra/traefik/dynamic/apps.yaml` +- Subdomain pattern: `.asepharyana.my.id` and `.asepharya.web.id` +- TLS certs from volume mounts (not auto-ACME) +- Middleware chain: `secure-headers` → `compress` → `retry` → `rate-limit` → `buffer` + +### Image Tagging +- `sha-` — immutable, for deterministic rollbacks +- `latest` — mutable, for convenience +- Registry: `ghcr.io/asepharyana/asepharyana-hub/` +- Build cache: `sha--buildcache` (registry-based caching) + +## Adding a New Service + +1. Create a separate repo for the app code +2. Add as submodule: `git submodule add apps/` +3. Create Dockerfile in `infra/docker/` +4. Create compose file in `infra/compose/` (app + Dapr sidecar) +5. Add Traefik router in `infra/traefik/dynamic/apps.yaml` +6. Add build job in `.github/workflows/docker-build-push.yml` +7. See `docs/add-new-app.md` for full guide + +## Commit Convention + +Format: `(): ` + +Types: `feat`, `fix`, `chore`, `docs`, `refactor`, `test`, `ci`, `perf`, `style` +Scopes: `scraper`, `infra`, `ci`, `dapr`, `nats`, `docs`, `deps`, `scripts`, `root` + +Scope is required. Use imperative mood. No period at end of subject line. Co-Authored-By footer for AI-generated commits. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 057527e..41caad8 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -600,6 +600,73 @@ docker system prune -a -f --- +--- + +## Proyek Ini: asepharyana-hub + +> Dokumentasi spesifik untuk repo ini. Lihat juga [ADR-0002](adr/0002-env-file-via-github-secret.md). + +### Topologi + +| Host | IP | Peran | +|------|----|-------| +| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr | +| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379) | + +### Environment Variables + +**Production `.env` tidak pernah di-commit.** File ini disimpan sebagai GitHub secret `ENV_FILE_PRODUCTION` dan di-SCP ke VPS saat deploy via `deploy-docker.yml`. + +Cara update: +```bash +# Baca current .env dari VPS +ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" + +# Update GitHub secret (dari output di atas) +cat > /tmp/env-updated << 'EOF' + +EOF +cat /tmp/env-updated | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub +``` + +**Jangan manual edit `.env` di VPS tanpa update GitHub secret juga** — nanti ke- overwrite pas deploy berikutnya. + +### Database + +| Variable | Value | +|----------|-------| +| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` | +| `REDIS_URL` | `redis://redis:6379` (Docker network) | + +### Kompose + +Proyek compose bernama `compose`, terdiri dari 5 file yang selalu di-include bersamaan: + +```bash +/root/asepharyana-hub/infra/compose/ +├── traefik.yml # Reverse proxy +├── shared.yml # Redis +├── nats.yml # NATS +├── dapr.yml # Dapr placement +└── scraper.yml # Scraper API +``` + +Perintah restart setelah update `.env` di VPS: +```bash +cd /root/asepharyana-hub +docker compose \ + -p compose \ + --env-file .env \ + -f infra/compose/traefik.yml \ + -f infra/compose/shared.yml \ + -f infra/compose/scraper.yml \ + -f infra/compose/nats.yml \ + -f infra/compose/dapr.yml \ + up -d --remove-orphans +``` + +--- + ## Checklist Deploy Proyek Baru 1. [ ] Dockerfile ditest lokal (`docker build`, `docker run`) diff --git a/docs/adr/0002-env-file-via-github-secret.md b/docs/adr/0002-env-file-via-github-secret.md new file mode 100644 index 0000000..ba491fb --- /dev/null +++ b/docs/adr/0002-env-file-via-github-secret.md @@ -0,0 +1,122 @@ +# ADR 0002: Production `.env` via GitHub Encrypted Secret + +## Status + +Accepted + +## Context + +The project runs on a remote VPS (`orangevps`, IP `45.127.35.244`) that hosts multiple services via Docker Compose. These services require environment variables (database credentials, API keys, tokens) that must not be committed to the repository. + +The production `.env` file on the VPS is **not** a copy of the committed `.env` in the repo root — it contains additional secrets (Portainer tokens, Discord bot tokens, etc.) that only exist in production. + +Previously, the `.env` file on the VPS was edited manually via SSH, which led to drift between the local `.env` and the production `.env`. When the database server IP or port changed in the local `.env`, the production `.env` was not updated, causing service outages. + +## Decision + +The production `.env` file is stored as a **GitHub Actions encrypted secret** named `ENV_FILE_PRODUCTION`. During deployment, the `.github/workflows/deploy-docker.yml` workflow writes this secret to a file and SCPs it to the VPS. + +### Flow + +``` +GitHub Secret (ENV_FILE_PRODUCTION) + │ + ▼ (deploy-docker.yml) +echo "$ENV_FILE_PRODUCTION" > .env.prod +scp .env.prod → VPS:$VPS_TARGET_DIR/.env + │ + ▼ (docker compose --env-file .env up) +Container reads $DATABASE_URL, $JWT_SECRET, etc. +``` + +### How to update + +```bash +# 1. Read current content from the VPS +ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" + +# 2. Pipe updated content to the GitHub secret +# (requires gh CLI with repo access) +cat /path/to/updated-env | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub + +# 3. Trigger a redeploy to push it to the VPS +gh workflow run deploy-docker.yml + +# OR apply immediately on the VPS (for hotfix): +ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.env" +# Then restart affected containers +``` + +## Server Topology + +| Host | IP | Role | +|------|----|------| +| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr | +| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379), Browserless | + +## Database + +| Variable | Value | +|----------|-------| +| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` | +| `REDIS_URL` | `redis://redis:6379` (Docker network, overridden per-service) | +| `EXTERNAL_BROWSERLESS_WS` | `ws://43.134.105.109:3001/?token=...` (external proxy) | + +> **Important:** The Docker Compose `environment:` section uses variable interpolation (`${DATABASE_URL}`), which is resolved from the `--env-file .env` at compose time — NOT from the service's `env_file`. Both must be kept in sync. + +## Docker Compose Project Structure + +The VPS runs a single Docker Compose project named `compose` composed of multiple files: + +```bash +/root/asepharyana-hub/infra/compose/ +├── traefik.yml # Reverse proxy (TLS termination, routing) +├── shared.yml # Redis +├── nats.yml # NATS message broker + JetStream +├── dapr.yml # Dapr placement service +└── scraper.yml # Scraper API + Dapr sidecar +``` + +All files are always included together for dependency resolution: + +```bash +docker compose \ + --env-file .env \ + -f infra/compose/traefik.yml \ + -f infra/compose/shared.yml \ + -f infra/compose/scraper.yml \ + -f infra/compose/nats.yml \ + -f infra/compose/dapr.yml \ + up -d +``` + +## GitHub Secrets Required + +| Secret | Description | +|--------|-------------| +| `SSH_PRIVATE_KEY` | SSH key for VPS access | +| `VPS_HOST` | `45.127.35.244` | +| `VPS_USER` | `root` | +| `VPS_TARGET_DIR` | `/root/asepharyana-hub` | +| `ENV_FILE_PRODUCTION` | Full `.env` content for production | + +## Consequences + +### Positive + +- Environment is version-controlled via GitHub Secrets audit log. +- No risk of committing secrets to the repo. +- Deployment is fully automated — `.env` is pushed on every deploy. +- Easy to rotate secrets: update `ENV_FILE_PRODUCTION` and redeploy. + +### Negative + +- The secret is opaque — you cannot diff it or review changes via PR. +- If the secret falls out of sync with the local `.env`, services silently break on next deploy. +- Requires `gh` CLI or GitHub UI to update — not a simple file edit. + +### Mitigations + +- Keep the **committed `.env`** in the repo root as the source of truth for non-secret values (database URL, ports, API endpoints). +- Document any manual SSH hotfix at the same time as updating the GitHub secret. +- Run `gh secret set ENV_FILE_PRODUCTION` with the latest server `.env` content after any hotfix.