144 Commits
Author SHA1 Message Date
asepharyanaandClaude Opus 5 21743cedfa chore(infra): remove llm-api service (retired from prod)
Deploy Infra Config / deploy-infra (push) Canceled after 0s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 18:53:29 +07:00
MythEclipse 63c3b87f39 fix(hamc): hermyhq port 4023→4024, add hermyhq-render worker unit
Deploy Infra Config / deploy-infra (push) Canceled after 0s
- hermyhq.service: PORT 4024 (4023 occupied by pr-agent-bun)
- Caddyfile.prod: hamc.* proxy → 4024
- hermyhq-render.service: new systemd unit for shorts render worker
  (Remotion + edge-tts, runs /opt/hermyhq-renderer)
2026-09-22 20:16:33 +07:00
asepharyana c5c0a961b0 chore(caddy): route pr-agent to Bun server (4023)
Deploy Infra Config / deploy-infra (push) Canceled after 0s
Cut-over pr-agent.asepharyana.my.id from the Python pr_agent server (4002)
to the Bun/TypeScript re-implementation (4023, pr-agent-bun.service).
Verified: webhook -> review -> LLM claude-opus-5 -> persistent comment
published (PR #19 nextjs-template).
2026-09-21 13:38:04 +07:00
asepharyana c25c645818 feat(dev): add dev.asepharyana.my.id Caddy route → 127.0.0.1:3000
Deploy Infra Config / deploy-infra (push) Canceled after 0s
Ephemeral dev preview of the Next.js template — no systemd service; dev
server runs on-demand (bun run dev), Caddy routes :3000, 502s when stopped.
2026-09-20 21:48:22 +07:00
MythEclipse 9e6078af9b feat(infra): add hamc.asepharyana.my.id (hermyhq) Caddy routing + systemd unit
Deploy Infra Config / deploy-infra (push) Canceled after 0s
- Caddy: hamc.asepharyana.my.id -> 127.0.0.1:4023 (import proxy)
- systemd: hermyhq.service (Next.js 16 standalone, profile /nix/var/nix/profiles/hermyhq, env /etc/hermyhq/hermyhq.env)
2026-09-20 15:51:34 +07:00
asepharyana bda852f287 feat(infra): add mcpedia.asepharyana.my.id Caddy block -> local mcpedia (wiki alias)
Deploy Infra Config / deploy-infra (push) Canceled after 0s
2026-09-02 12:31:18 +07:00
asepharyana 754bc2876f fix(caddy): serve gmw.asepharyana.my.id → :4009 (was 525)
Deploy Infra Config / deploy-infra (push) Canceled after 0s
GMW dashboard SSL was broken (Cloudflare 525) because gmw was not in
the Caddyfile — no Let's Encrypt cert / TLS origin for the hostname.
Add gmw.asepharyana.my.id block proxying to gmw-proxy nginx (4009),
same as imphnen. Public dashboard now reachable (200, valid cert).
2026-09-01 20:19:00 +07:00
asepharyana b37daa52b0 chore(infra): remove tools app — Caddy route, README, add-new-app docs
Deploy Infra Config / deploy-infra (push) Canceled after 0s
Project tools dihapus (2026-08-28): GitHub repo + local source + VPS deploy
(3 systemd units + Nix profiles) sudah dihapus. Caddy route
tools.asepharyana.my.id removed; README + docs updated.
ADR 0003 dibiarkan sebagai catatan historis rename.
2026-08-27 21:17:05 +07:00
asepharyana 56cee21e66 ci(infra): sync full infra config (caddy + systemd drop-ins + prometheus), rename workflow to infra-deploy 2026-08-27 20:43:06 +07:00
asepharyana 6ac562a869 ci: validate Caddyfile with real caddy binary instead of naive brace count
Deploy Caddy Config / deploy-caddy (push) Canceled after 0s
2026-08-27 19:50:24 +07:00
Asep Haryana 4c6c368764 merge: integrate remote dependabot/submodule bump history into infra-only repo 2026-08-27 19:44:55 +07:00
Asep Haryana 0194a865b7 refactor(infra): repurpose asepharyana-hub into infra-only reverse-proxy repo
- Remove app submodules (hub/scraper/tools/llm-api/plugins) — apps are now
  standalone repos with their own flake.nix + deploy.yml CI
- Delete monorepo CI: nix-build.yml matrix, update-submodule.yml, lint, security,
  flakehub-publish — replaced by infra-only caddy-deploy.yml
- Sync Caddyfile.prod with live /etc/caddy/Caddyfile (add wiki. + mcp. blocks)
- Prune legacy Docker/Traefik/Dapr/NATS/otel + scripts/root tooling
- Docs: rename ADR 0001 superseded, add ADR 0003 (repo rename + split CI),
  update add-new-app, infra README, troubleshooting
2026-08-27 19:44:10 +07:00
dependabot[bot]andAsep Hariyana 1f490cf9c2 chore(deps-dev): bump @biomejs/biome from 2.5.3 to 2.5.10 in the development group across 1 directory (#12)
* chore(deps-dev): bump @biomejs/biome

Bumps the development group with 1 update in the / directory: [@biomejs/biome](https://github.com/biomejs/biome/tree/HEAD/packages/@biomejs/biome).


Updates `@biomejs/biome` from 2.5.3 to 2.5.10
- [Release notes](https://github.com/biomejs/biome/releases)
- [Changelog](https://github.com/biomejs/biome/blob/main/packages/@biomejs/biome/CHANGELOG.md)
- [Commits](https://github.com/biomejs/biome/commits/@biomejs/biome@2.5.10/packages/@biomejs/biome)

---
updated-dependencies:
- dependency-name: "@biomejs/biome"
  dependency-version: 2.5.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: sync lockfile for dependabot update

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Asep Hariyana <asepharyana.id@gmail.com>
2026-08-26 09:12:28 +07:00
monrepo-bot 851e82d625 chore: update scraper-api to 7ff47a79bcb4 2026-08-26 01:55:19 +00:00
monrepo-bot 8fe5a478b5 chore: update hub to 7b00c9deb2bc 2026-08-26 01:54:35 +00:00
asepharyana a2e65e3980 chore: update dependabot-auto-merge.yml 2026-08-26 08:53:32 +07:00
asepharyana 1bbba46b9a chore: update dependabot.yml 2026-08-26 08:53:30 +07:00
monrepo-bot 80ea5f0b2e chore: update hub to d3ae61b0c556 2026-08-25 13:33:16 +00:00
monrepo-bot 6cc583d3b9 chore: update scraper-api to c709c84a4d42 2026-08-25 13:33:10 +00:00
asepharyana 912fd6864f feat: add dependabot auto-merge workflow 2026-08-25 20:32:53 +07:00
monrepo-bot eba29a2fa1 chore: update hub to eb7b8a12fdd4 2026-08-25 13:22:06 +00:00
monrepo-bot 76df15616e chore: update hub to fab67538e15b 2026-08-25 11:20:26 +00:00
monrepo-bot 07a81df64d chore: update hub to 0de569cff475 2026-08-25 11:18:01 +00:00
monrepo-bot bbbb1d2cee chore: update scraper-api to 55cfd513e3f8 2026-08-25 11:17:49 +00:00
monrepo-bot f9d8445443 chore: update llm-api to 1916ebe40bcb 2026-08-25 11:17:38 +00:00
monrepo-bot 4f3442f97b chore: update tools to b35b00376dd1 2026-08-25 11:17:18 +00:00
asepharyana 93cf1ffb9b feat: update dependabot config 2026-08-25 18:16:34 +07:00
monrepo-bot 499721c5e3 chore: update hub to ac5ca405dda4 2026-08-25 11:16:15 +00:00
asepharyana 09c5311568 feat: update dependabot config 2026-08-25 18:12:46 +07:00
monrepo-bot e74a303b9d chore: update scraper-api to a23698a6c4ef 2026-08-25 11:08:57 +00:00
monrepo-bot 0bcd6582f4 chore: update hub to a0042942412a 2026-08-25 00:11:51 +00:00
monrepo-bot a38a99b6f4 chore: update hub to bc7279646715 2026-08-25 00:11:43 +00:00
monrepo-bot 53cc225bff chore: update hub to 90267abb9220 2026-08-25 00:11:35 +00:00
monrepo-bot a20b0be8da chore: update hub to 7e44d677aca4 2026-08-24 00:23:43 +00:00
monrepo-bot 76ccf5488b chore: update hub to d5cd4ece013b 2026-08-24 00:17:46 +00:00
monrepo-bot 29271ccc05 chore: update hub to e5fb73f61eba 2026-08-21 00:11:29 +00:00
monrepo-bot 6a1d199443 chore: update hub to 4a6b498879fa 2026-08-20 00:11:52 +00:00
monrepo-bot eb19385ab4 chore: update hub to ed9fa16b5760 2026-08-17 00:18:19 +00:00
monrepo-bot a1c0ef25ef chore: update hub to 1640230c66d3 2026-08-17 00:18:09 +00:00
monrepo-bot 609da6b6bc chore: update hub to c891439857df 2026-08-14 00:14:22 +00:00
monrepo-bot 402499d109 chore: update hub to bfbfc175ca82 2026-08-13 00:13:26 +00:00
monrepo-bot 1389d5faa3 chore: update hub to b30e6936b1df 2026-08-11 00:12:50 +00:00
monrepo-bot 59e6c5c6a1 chore: update hub to 9ea002a7a10d 2026-08-10 00:24:27 +00:00
monrepo-bot 4f8a975337 chore: update hub to e67db071bb0f 2026-08-10 00:24:15 +00:00
monrepo-bot c251867f05 chore: update hub to 17a785c9d277 2026-08-10 00:18:32 +00:00
monrepo-bot b06cddca4a chore: update hub to 915d8482947a 2026-08-10 00:18:27 +00:00
monrepo-bot 19aface599 chore: update hub to a5bf846e52fd 2026-08-10 00:18:19 +00:00
monrepo-bot 528a109d53 chore: update hub to eb80d4a0fd73 2026-08-07 00:10:49 +00:00
asepharyana 97b635e8c6 firewall: whitelist TCPShield proxy ranges untuk port 25565 (Minecraft) 2026-08-06 20:42:39 +07:00
mytheclipsebotreview[bot] 99e438d9c5 Auto-merge PR #10
chore(deps): bump actions/checkout from 6 to 7 in the github-actions group
2026-08-06 13:28:09 +00:00
dependabot[bot] 8e850f05c4 chore(deps): bump actions/checkout in the github-actions group
Bumps the github-actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-06 13:16:00 +00:00
asepharyana 6c2a4fc9d2 fix(infra): sync Caddyfile.prod with live — add dashboard.asepharyana.my.id block
dashboard was proxied via wildcard but missing from live Caddy site
blocks -> Cloudflare 525. Live /etc/caddy/Caddyfile is now the
authoritative copy; repo reference updated to match byte-for-byte so
the drift cannot recur.
2026-08-05 09:44:26 +07:00
Asep Haryana 3e92f68d78 fix(infra): revert imphnen block ke gmw-proxy 4009 (dimentorin gak dipublish via domain ini) 2026-08-04 23:12:02 +07:00
Asep Haryana 2050129062 fix(infra): imphnen site block — SPA :5173 + API /v1 → :4099 (was 525: no block on live caddy) 2026-08-04 22:51:09 +07:00
asepharyana a6148753ca fix(infra): 9router Caddy timeouts — false 504 on slow LLM TTFT
9router combo models (deepseek-v4-flash-free fallback) have TTFT up to
30-40s. Default (proxy) response_header_timeout 30s / read 60s caused
false-positive 504 'timeout awaiting response headers' even while 9router
was processing. Give the 9router route dedicated long timeouts
(response_header 120s, read/write 300s). Syncs /etc/caddy/Caddyfile.
2026-08-04 22:02:09 +07:00
asepharyana d02989f1a7 ci: add Nix GC cleanup job on VPS after deploy 2026-08-04 13:58:10 +07:00
monrepo-bot 6e317eb05a chore: update hub to 55310768a5ac 2026-08-04 02:39:29 +00:00
asepharyana 4e114edb89 chore: update settings.json with new hooks and commands; update .gitignore to include .serena/ directory 2026-08-03 16:49:28 +07:00
aseph 49123c08b3 ci: use free GHA Nix cache (disable FlakeHub cache, not subscribed) 2026-08-03 16:44:32 +07:00
asepharyana 539b5b8320 ci: enable FlakeHub Cache (id-token: write + use-flakehub) 2026-08-03 16:24:48 +07:00
monrepo-bot 39df52aab7 chore: update hub to ad6b1847c563 2026-08-03 09:23:16 +00:00
Asep Haryana b326911478 chore(infra): bump hub flake rev to a90d0c4 (error rate fix) 2026-08-03 13:07:38 +07:00
monrepo-bot 8242325ffd chore: update hub to a90d0c43336a 2026-08-03 06:06:50 +00:00
Asep Haryana c075cf282b fix(infra): correct hub flake rev to 62966fd (full sha) 2026-08-03 12:48:14 +07:00
Asep Haryana 93936efd40 chore(infra): bump hub flake rev to 62966fd (LLM dashboard card) 2026-08-03 12:46:11 +07:00
monrepo-bot 5f66cfc12b chore: update hub to 62966fd7cf45 2026-08-03 05:45:45 +00:00
Asep Haryana cc8c3f1bf3 chore(infra): add llm-api to prometheus targets.yml (now exposes /metrics) 2026-08-03 11:56:35 +07:00
Asep Haryana 0c36007da4 chore(infra): bump llm-api flake rev to 5f7ead5 2026-08-03 11:46:08 +07:00
monrepo-bot b7d02647d9 chore: update llm-api to 5f7ead550308 2026-08-03 04:45:04 +00:00
Asep HaryanaandClaude Opus 5 ade2798ded chore(infra): bump llm-api flake rev to 7cec411
Fixes streaming reasoning/content split (robust boundary detection,
flush un-tagged output) and renders reasoning in the chat UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 11:04:18 +07:00
monrepo-bot ef14ca431f chore: update llm-api to 7cec411cba7c 2026-08-03 04:04:04 +00:00
Asep HaryanaandClaude Opus 5 f681e624b3 chore(infra): bump llm-api flake rev to b636496
Deploys the generation-flow refactor (unified generate, streaming fixes,
tool-calling, strict model validation, AddBos::Never).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:50:23 +07:00
Asep HaryanaandClaude Opus 5 fdbdcbc1ec fix(ci): trigger nix deploy on every push to main
GitHub path filters do not match submodule gitlink changes, so the
`paths: apps/**` filter meant a submodule pointer update never triggered
the deploy. Drop the filter so any push to main deploys (matches the
documented "Push to main -> nix build -> systemctl restart").

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:35:56 +07:00
Asep HaryanaandClaude Opus 5 b7b60e9125 chore(ci): remove legacy Docker workflows (moved to Nix)
Docker was decommissioned 2026-08-02 but docker-build-push.yml and
deploy-docker.yml were left behind. They still listened to
repository_dispatch: [submodule-updated], so every app push queued a
redundant Docker build alongside the intended Nix deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:13:38 +07:00
monrepo-bot 722709f6fb chore: update llm-api to b63649649758 2026-08-03 01:58:50 +00:00
Asep Haryana d20643fbbd fix(infra): ship next.config.ts into Nix store for hub
next start re-reads next.config.ts at runtime; the OLD store had only
.next/public/package.json/node_modules, so images.maximumDiskCacheSize
fell back to defaults (undefined) and the image-optimizer disk LRU did
mkdir(<store>/.next/cache/images) -> EACCES on every optimized image.
Also now that next.config.ts sets maximumDiskCacheSize:0 +
isrFlushToDisk:false, shipping the file makes runtime match the baked
build config, eliminating the EACCES spike.
2026-08-03 04:33:46 +07:00
Asep Haryana 7da1727603 fix(firewall): accept IPv6 MLD/ND multicast (ff02::1, ff02::2, ff02::fb)
Neighbor multicast to ff02::1 (MLDv2 reports from other hosts) was
hitting the LOG+DROP tail — ~1800 FW6-DROP lines/6h in journald,
i.e. ~5/min of pure log spam from ICMPv6 layer-2 discovery. Accept
link-local multicast ranges before the logging rule.
2026-08-03 04:06:32 +07:00
Asep Haryana 9e9fd229d0 chore(infra): bump hub flake rev to 6829c7e (image disk cache fix) 2026-08-03 04:05:23 +07:00
Asep Haryana 40095f75f4 chore(infra): bump hub submodule to 6829c7efe2a735d249f5135bff4a0fc28411a295 (disable Next image disk cache) 2026-08-03 04:05:15 +07:00
asepharyana 3781a8882d chore(infra): scraper OTEL env drop-in (OTLP endpoint 127.0.0.1:4317) 2026-08-02 21:05:20 +07:00
asepharyana da2f4a2f2c chore(infra): prometheus file_sd targets for nix services (gmw-backend/api-metrics) 2026-08-02 21:05:12 +07:00
asepharyana ad22a39a9e chore(infra): use full sha for hub flake rev 2026-08-02 20:56:16 +07:00
asepharyana 2f969972a1 chore(infra): bump hub flake rev to f35edb0 (dashboard fix) 2026-08-02 20:55:55 +07:00
monrepo-bot 620f06fbe0 chore: update hub to f35edb00d726 2026-08-02 13:55:07 +00:00
asepharyana e6033c8b12 feat(infra): hardening firewall + sysctl + ssh
- deny-by-default iptables/ip6tables: public cuma 22/80/443/4013,
  sisanya localhost + tailscale CGNAT; log FW-DROP rate-limited
- persist via netfilter-persistent (rules.v4/v6) + /usr/local/bin/firewall.sh
- sysctl hardening: syncookies, rp_filter, redirect off, kptr/dmesg
  restrict, protected links, somaxconn, tcp retries tuner
- sshd: PasswordAuthentication no (fix 50-cloud-init.conf override),
  PermitRootLogin prohibit-password, MaxAuthTries 4, LoginGraceTime 30,
  X11Forwarding no
2026-08-02 20:22:49 +07:00
asepharyana 500af023de fix(infra): otel collector no self-loop trace export
- hapus otlp/jaeger exporter (jaeger mati sejak Docker dihapus) yang
  nunjuk ke 127.0.0.1:4317 (port receiver sendiri) -> loop trace tanpa
  henti, ~1.8jt baris log/hari di journal
- traces pipeline -> nop exporter, metrics -> prometheus
- logs pipeline dihapus (tidak ada konsumen)
2026-08-02 19:59:31 +07:00
Asep Haryana e280d196f9 chore: update llm-api to 344bc195fa95 2026-08-02 17:02:59 +07:00
Asep Haryana 4927338c98 docs: sync infra docs to Caddy+Nix 4000s (Traefik/Docker legacy) 2026-08-02 16:54:17 +07:00
monrepo-bot 54ee903647 chore: update llm-api to 344bc195fa95 2026-08-02 09:45:06 +00:00
Asep Haryana 35cbcdcc92 chore(infra): add tuned Caddyfile.prod reference 2026-08-02 16:37:05 +07:00
monrepo-bot 324fe08913 chore: update llm-api to 81c51772495c 2026-08-02 09:18:56 +00:00
Asep Haryana e171201b1c chore: sync port references and docs to 4000s infra 2026-08-02 16:16:40 +07:00
Asep Haryana ec1394bdc5 chore: bump tools submodule (ports 4007/4008) 2026-08-02 14:31:47 +07:00
asepharyana f0787b5f23 fix(nix): restrict flake to x86_64-linux (nixpkgs 26.11 dropped darwin) 2026-08-01 18:03:44 +07:00
asepharyana 0df6c35d5a ci: publish flake to FlakeHub (rolling) 2026-08-01 17:58:22 +07:00
Asep Haryana 5d21848e87 perf(infra): traefik response speed - HTTP/3 + backend conn pooling
- entryPoints.websecure.http3: enable QUIC (UDP 443, alt-svc h3)
- serversTransport maxIdleConnsPerHost 2->100: reuse backend keep-alive
- forwardingTimeouts.dialTimeout 30s->3s: fail fast on dead backends
- disable version-check/anonymous-usage network chatter
2026-08-01 13:12:50 +07:00
Asep Haryana b96548bffa fix(infra): raise upload-buffer response cap to 2GB
Chunked file streaming (TeleUploader) serves multi-part bodies up to
120MB+; 10MB maxResponseBodyBytes made every big download 500.
2026-08-01 12:52:34 +07:00
Asep Haryana e19fa76dc3 fix(infra): add teleuploader traefik router for upload hostnames
Restore upload.asepharyana.my.id/.web.id routing lost during Nix migration
(docker label router removed with container). Upload chain uses 2GB body
buffer + dedicated 300/100 rate limit, service targets bun on host:3000.
2026-08-01 12:34:22 +07:00
Asep Haryana c535ee7858 ci(otel): add logs pipeline to otel-collector (OTLP /v1/logs 404 fix) 2026-07-31 14:13:56 +07:00
Asep Haryana 80b5161630 ci(infra): traefik dynamic config path -> /home/code (drop /root dependency) 2026-07-31 14:02:07 +07:00
Asep Haryana 1ccc0355ac ci(traefik): add hermes dashboard route via file provider 2026-07-31 13:50:46 +07:00
asepharyana 12aed642a9 ci(traefik): add zeavis routes via file provider (Nix migration) 2026-07-31 12:42:14 +07:00
asepharyana f8e3dccb44 ci(traefik): lidm routes -> host ports 3100/3101 (3000/3001 occupied by teleuploader+gmw) 2026-07-31 11:37:13 +07:00
asepharyana 1811c6adc2 ci(traefik): add lidm frontend+backend routes via file provider (Nix migration) 2026-07-31 11:29:47 +07:00
dependabot[bot] ceeac5c02f chore(deps): bump the github-actions group across 1 directory with 3 updates (#9)
Bumps the github-actions group with 3 updates in the / directory: [actions/checkout](https://github.com/actions/checkout), [DeterminateSystems/nix-installer-action](https://github.com/determinatesystems/nix-installer-action) and [DeterminateSystems/magic-nix-cache-action](https://github.com/determinatesystems/magic-nix-cache-action).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v7)

Updates `DeterminateSystems/nix-installer-action` from 16 to 22
- [Release notes](https://github.com/determinatesystems/nix-installer-action/releases)
- [Commits](https://github.com/determinatesystems/nix-installer-action/compare/v16...v22)

Updates `DeterminateSystems/magic-nix-cache-action` from 8 to 14
- [Release notes](https://github.com/determinatesystems/magic-nix-cache-action/releases)
- [Commits](https://github.com/determinatesystems/magic-nix-cache-action/compare/v8...v14)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: DeterminateSystems/magic-nix-cache-action
  dependency-version: '14'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: DeterminateSystems/nix-installer-action
  dependency-version: '22'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-30 22:34:36 +07:00
Asep Haryana d0c9f16d5d chore(infra): disable Docker compose for Nix-migrated services (hub, scraper, tools, llm-api) 2026-07-30 22:29:20 +07:00
Asep Haryana 6cbafbfe2e fix(flake): update scraper pinned rev for config fix 2026-07-30 22:28:24 +07:00
GitHub Action d6b9a0806b chore: update manifests and submodules [skip ci] 2026-07-30 15:23:34 +00:00
Asep Haryana 60a1af436f fix(infra): llm-api port 8080→8082 (conflict with gmw-proxy nginx) 2026-07-30 22:18:56 +07:00
Asep Haryana 2459541677 fix(scraper): update submodule - config list_separator fix 2026-07-30 22:18:03 +07:00
Asep Haryana 33b999d2b7 fix(ci): disable Determinate Nix / FlakeHub (no flakehub flakes used)
DeterminateSystems/nix-installer-action defaults to determinate:true
which tries to auth with FlakeHub via GitHub JWT. We don't use any
FlakeHub flakes, so disable it — skipping the auth entirely.
2026-07-30 22:05:41 +07:00
Asep Haryana 59f131f951 fix(ci): combine build+deploy per-service, pass exact store path
Separate deploy job failed because it used ls to find store paths
by name, finding OLD local paths instead of the freshly copied CI
paths. Now each service builds, copies, and updates its profile
in a single job using the exact store path from the build output.
No more guessing which path is the right one.
2026-07-30 21:42:39 +07:00
Asep Haryana cc8c5088a4 fix(ci): use SSH_PRIVATE_KEY secret, sanitize key format
- Ganti secret name: VPS_SSH_KEY → SSH_PRIVATE_KEY (nama yg ada)
- Fix nix copy URL: ***@ → $VPS_USER@
- Sanitize SSH key: strip \r\n, validasi dengan ssh-keygen
- Cegah libcrypto error dari key format broken
2026-07-30 21:32:40 +07:00
Asep Haryana d15e8621a9 fix(ci): proper SSH user in nix copy, deploy job structure
- Fix nix copy URL: ***@ → $VPS_USER@
- Store path from build output, passed across jobs
- Deploy job waits for all builds via needs: build
- SSH key setup in its own step, guarded by main branch
- Only deploy on main branch pushes
- Remote deploy script fetches from VPS nix store
2026-07-30 21:14:20 +07:00
Asep Haryana 33d5b12ec6 fix: tools-workers install path (no cd backend in installPhase) 2026-07-30 20:11:31 +07:00
Asep Haryana b5596e1398 feat(infra): full Nix migration — all 6 services + CI/CD
- flake.nix: 6 derivations (hub, scraper, tools-gateway, tools-workers, tools-frontend, llm-api)
- Fetch submodule source via builtins.fetchGit with pinned revs
- Fix cargo HOME/TMPDIR for Nix sandbox permission issues
- Fix llm-api: CMake/Clang deps for llama.cpp-sys2 bindgen
- Add LIBCLANG_PATH, LD_LIBRARY_PATH for Rust bindgen builds
- Systemd units: tools-gateway (3501), tools-frontend (3500), tools-workers, llm-api (8080)
- tools.target for grouped management
- Env configs: /etc/tools/env, /etc/llm-api/env
- GitHub Actions: nix-build.yml — matrix build + nix copy + deploy
- Update Traefik apps.yaml: tools/host.docker.internal:3500, llm-api/host.docker.internal:8080
- iptables: allow Docker→host on 3099, 4091, 3500, 3501, 8080
- Add scripts/nix-deploy.sh for CI/CD deploy step
2026-07-30 19:45:43 +07:00
Asep Haryana 46730ec07d feat(infra): Nix build for scraper, GitHub Actions workflow
- Build scraper (Rust) with Nix — cargo build --release
- Create scraper systemd unit (port 4091), env from Docker config
- Fix HOME/CARGO_HOME for Rust/cargo in Nix sandbox
- Update Traefik apps.yaml: scraper -> host.docker.internal:4091
- Add iptables rules for port 4091 (Docker->host)
- Add GitHub Actions workflow: nix-build.yml (determinate-nix + deploy)
- Save iptables rules persistently
2026-07-30 18:46:45 +07:00
Asep Haryana 1ae8a53cd4 chore: ignore Nix build result symlink 2026-07-30 18:33:52 +07:00
Asep Haryana ff52f8841c feat(infra): Nix build for hub app, systemd deployment
- Add flake.nix with derivations for hub (Next.js), scraper, tools, llm-api
- Create hub systemd unit (port 3099)
- Update Traefik dynamic config to point to host hub service
- Add iptables rule for Docker-to-host communication
- Use fetchGit for submodule source resolution
2026-07-30 18:33:43 +07:00
monrepo-bot b747ae5c3b chore: update hub to b007ced615dc 2026-07-29 00:12:03 +00:00
monrepo-bot 64ce812b4e chore: update hub to 5370fd47954d 2026-07-29 00:10:06 +00:00
monrepo-bot 9837d9dc50 chore: update hub to 407aa7b8993b 2026-07-28 00:11:32 +00:00
monrepo-bot 86dda74174 chore: update hub to a803849cba6c 2026-07-27 00:21:20 +00:00
monrepo-bot d045602f3e chore: update hub to 6042a6056a99 2026-07-27 00:17:20 +00:00
GitHub Action 32049c70ac chore: update manifests and submodules [skip ci] 2026-07-26 12:18:15 +00:00
asepharyana be7348a27d fix(infra): remove buffer/compress from llm-api for real SSE streaming
- llm-api uses llm-chain instead of common-chain
- llm-chain excludes buffer & compress middlewares
- Prevents Traefik from buffering streaming SSE responses
- Each token is now flushed immediately
2026-07-26 19:14:52 +07:00
asepharyana 43f0df4493 fix(infra): remove buffer/compress from llm-api middleware chain
- Buffer middleware causes SSE stream buffering
- Compress middleware breaks real-time streaming
- Added llm-chain with only secure-headers + retry + rate-limit
- llm-api router now uses llm-chain@file instead of common-chain@file
2026-07-26 19:13:08 +07:00
GitHub Action f0c7534152 chore: update manifests and submodules [skip ci] 2026-07-26 12:09:58 +00:00
asepharyana a4687c9d5b fix(llm-api): update submodule - reasoning/content stream separation 2026-07-26 19:04:36 +07:00
GitHub Action e42e967b7f chore: update manifests and submodules [skip ci] 2026-07-26 11:27:08 +00:00
asepharyana d99bc06df3 fix(llm-api): update submodule to 6ff31b5 — Jinja template via minijinja
- Replaced llama-cpp-2 apply_chat_template with minijinja rendering
- reasoning_content separate from content in API response
- Works with MiniCPM5 thinking model natively
2026-07-26 18:21:43 +07:00
GitHub Action 87df0d7ae2 chore: update manifests and submodules [skip ci] 2026-07-26 10:15:32 +00:00
monrepo-bot 518afa5645 chore: update llm-api to 9a63ff16018f 2026-07-26 10:11:01 +00:00
GitHub Action 2913f515f0 chore: update manifests and submodules [skip ci] 2026-07-26 09:14:25 +00:00
monrepo-bot 05e20df680 chore: update llm-api to 59c77108a55b 2026-07-26 09:09:23 +00:00
GitHub Action 59e2f9d173 chore: update manifests and submodules [skip ci] 2026-07-26 09:08:26 +00:00
monrepo-bot f78b826e59 chore: update llm-api to 495b9ed126e6 2026-07-26 09:03:36 +00:00
GitHub Action d75424634d chore: update manifests and submodules [skip ci] 2026-07-26 08:58:06 +00:00
GitHub Action cc8cefb33a chore: update manifests and submodules [skip ci] 2026-07-26 08:53:27 +00:00
monrepo-bot 757b48b446 chore: update llm-api to 254532458b16 2026-07-26 08:49:23 +00:00
monrepo-bot 20c81b4018 chore: update llm-api to 7c8f747faf43 2026-07-26 08:49:00 +00:00
GitHub Action ca81a8b7ee chore: update manifests and submodules [skip ci] 2026-07-26 08:44:28 +00:00
GitHub Action ac25a51113 chore: update manifests and submodules [skip ci] 2026-07-26 08:42:08 +00:00
asepharyana 10aeda4d4b fix: update llm-api MODEL_PATH for new MiniCPM5 GGUF
Updates MODEL_PATH to MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-Q8_0.gguf
2026-07-26 15:40:14 +07:00
95 changed files with 737 additions and 9566 deletions
-1
View File
@@ -1 +0,0 @@
1.3.11
-29
View File
@@ -1,29 +0,0 @@
{
"skills": [
{
"name": "clean-code",
"filePattern": ".claude/skills/clean-code/SKILL.md",
"description": "Clean Code, Clean Architecture, SOLID, TDD — dari kana-best-practice-engineering"
},
{
"name": "hub-rules",
"filePattern": ".claude/skills/hub-rules.md",
"description": "Aturan repository hub, submodule, infra patterns, dan arsitektur"
},
{
"name": "commit-convention",
"filePattern": ".claude/skills/commit-convention.md",
"description": "Commit message convention — type(scope): description"
},
{
"name": "event-driven",
"filePattern": ".claude/skills/event-driven.md",
"description": "Event-driven patterns with Dapr + NATS untuk hub services"
},
{
"name": "deploy-workflow",
"filePattern": ".claude/skills/deploy-workflow.md",
"description": "CI/CD pipeline, Docker patterns, deployment guide"
}
]
}
-1
View File
@@ -1 +0,0 @@
/home/asephs/kana-best-practice-engineering/skills/clean-code
-65
View File
@@ -1,65 +0,0 @@
---
name: commit-convention
description: Enforce commit message convention untuk Asepharyana Hub
---
# Commit Convention — Asepharyana Hub
## Format
```
<type>(<scope>): <description>
[optional body]
[optional footer]
```
## Types
| Type | Usage |
| ---------- | ------------------------------------ |
| `feat` | Fitur baru |
| `fix` | Bug fix |
| `chore` | Maintenance, config, tooling |
| `docs` | Dokumentasi |
| `refactor` | Perubahan kode tanpa fungsional baru |
| `test` | Nambah/update test |
| `ci` | CI/CD workflows |
| `perf` | Optimasi performa |
| `style` | Formatting (tanda kutip, dll) |
## Scopes
| Scope | Area |
| ------------- | --------------------------------- |
| `scraper` | apps/scraper submodule |
| `infra` | infra/ (compose, traefik, docker) |
| `ci` | .github/workflows/ |
| `dapr` | Dapr config & sidecar |
| `nats` | NATS message bus |
| `docs` | Dokumentasi |
| `deps` | Dependencies |
| `scripts` | Utility scripts |
| `root` | Root config files |
## Contoh
```
feat(scraper): add anime detail caching via Dapr pubsub
fix(infra): correct NATS CLI flags for JetStream
chore(deps): update biome to v2.5.3
docs(infra): add deployment order for Dapr services
ci(deploy): add nats.yml to ALL_COMPOSE_FILES
refactor(scraper): migrate EventBus from tokio broadcast to Dapr pubsub
```
## Aturan
1. **Wajib** menyertakan scope dalam tanda kurung
2. **Wajib** `Co-Authored-By` untuk commit yang digenerate AI
3. **Gunakan imperative mood**: "add" bukan "added" / "adds"
4. **Jangan capitalize** type: `feat:` bukan `Feat:`
5. **No period** di akhir subject baris
6. Body explain **why** dan **what**, bukan **how**
7. Refer issue dengan `Closes #123` atau `Fixes #123` di footer
-103
View File
@@ -1,103 +0,0 @@
---
name: deploy-workflow
description: Panduan deploy, CI/CD, dan Docker compose patterns untuk Asepharyana Hub
---
# Deploy & Workflow — Asepharyana Hub
## CI/CD Pipeline
### Build Pipeline (`docker-build-push.yml`)
Trigger: push ke `main` yang touch `apps/**`, `infra/**`, `infra/docker/**`
1. **changes** — detect service mana yg berubah via git diff
2. **wait-submodule-ref** — (repository_dispatch only) tunggu SHA commit fetchable
3. **build** — matrix build per service, push ke GHCR (`sha-<short>` + `latest`)
4. **update-manifest** — update image tag di compose file, commit + push
### Deploy Pipeline (`deploy-docker.yml`)
Trigger: build selesai, atau push ke `main` touch `infra/**`
1. SSH ke `orangevps` (via `secrets.VPS_HOST`)
2. Sync repo (`git fetch --depth=1 + reset`)
3. Login ke GHCR
4. Deteksi compose file yg berubah
5. Pull images + restart container selektif
### Secrets Required
| Secret | Untuk |
|--------|-------|
| `SSH_PRIVATE_KEY` | SSH ke VPS |
| `VPS_HOST` | IP/host VPS (tailscale IP) |
| `VPS_USER` | SSH user, biasanya `root` |
| `VPS_TARGET_DIR` | Lokasi repo di VPS |
| `ENV_FILE_PRODUCTION` | .env content untuk production |
### Selective Deployment
- Hanya compose file yg berubah yang di-redeploy
- Selective: `UP_FLAGS="-d"` (tanpa `--remove-orphans`)
- Full deploy: `UP_FLAGS="-d --remove-orphans"`
## Docker Patterns
### Build dengan cargo-chef (Rust)
```dockerfile
FROM lukemathwalker/cargo-chef:latest-rust-1.89.0 AS chef
WORKDIR /app
FROM chef AS planner
COPY apps/scraper .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY apps/scraper .
RUN cargo build --release
```
### Runtime minimal untuk Rust binary
```dockerfile
FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3 && rm -rf /var/lib/apt/lists/*
```
## Image Tagging
- `sha-<short-sha>` — immutable, untuk rollback
- `latest` — mutable, untuk convenience
- Build cache: `sha-<short>-buildcache`
- Registry: `ghcr.io/asepharyana/asepharyana-hub/<service>`
## Manual Deploy Steps
```bash
# 1. Login GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u asepharyana --password-stdin
# 2. Full stack
docker compose -f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env up -d --remove-orphans
# 3. Selective (hanya satu service)
docker compose -f infra/compose/scraper.yml --env-file .env up -d
```
## Troubleshooting
### Container reach Tailscale
Pastikan route ke Tailscale di main table:
```bash
ip route add 100.64.0.0/10 dev tailscale0 table main
systemctl restart tailscale-routes
```
### Healthcheck gagal di scratch images
NATS dan Dapr placement pake scratch — tidak bisa healthcheck. Cukup `service_started` di depends_on.
### Dapr sidecar crash
```bash
docker logs scraper-api-dapr | grep -iE "fatal|error"
```
Penyebab umum: komponen config salah, NATS/Dapr placement belum siap.
-133
View File
@@ -1,133 +0,0 @@
---
name: event-driven
description: Event-driven patterns dengan Dapr + NATS untuk Asepharyana Hub
---
# Event-Driven Architecture — Asepharyana Hub
## Stack
- **Message Backbone**: NATS + JetStream (untuk streaming & job queue)
- **Pub/Sub Runtime**: Dapr sidecar per service (pubsub via Redis built-in)
- **State Store**: Dapr → Redis
## Event Topics Convention
```
hub.<domain>.<action>
Contoh:
hub.image.cached → Image selesai di-cache ke CDN
hub.image.repaired → Image diperbaiki (CNAME change)
hub.scrape.anime.done → Scrape anime selesai
hub.system.alert → Error/alert dari service
```
## CloudEvents Format
```json
{
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"subject": "anime-poster",
"id": "uuid-v4",
"time": "2026-07-21T10:00:00Z",
"datacontenttype": "application/json",
"data": { ... }
}
```
## Publish Event (Rust via HTTP API)
Gunakan `reqwest` langsung ke Dapr sidecar (SDK Rust masih experimental):
```rust
let event = serde_json::json!({
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"id": Uuid::new_v4().to_string(),
"time": chrono::Utc::now().to_rfc3339(),
"datacontenttype": "application/json",
"data": { "original_url": url, "cdn_url": cdn_url }
});
reqwest::Client::new()
.post("http://localhost:3500/v1.0/publish/pubsub/hub.image.cached")
.json(&event)
.send()
.await?;
```
## Service Invocation
```bash
curl http://localhost:3500/v1.0/invoke/<app-id>/method/<path>
```
## State Store
```bash
# Set
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
# Get
curl http://localhost:3500/v1.0/state/statestore/mykey
# Delete
curl -X DELETE http://localhost:3500/v1.0/state/statestore/mykey
```
## Scraper Event Integration
File yang perlu dimodifikasi untuk event-driven:
| File | Perubahan |
|------|-----------|
| `src/events/bus.rs` | Ganti backend dari tokio broadcast ke Dapr pub/sub |
| `src/bootstrap/mod.rs` | Init DaprClient, inject ke AppState |
| `src/presentation/state.rs` | Tambah `dapr_client` field |
| `src/proxy/use_cases.rs` | Publish `ImageRepaired` & `ImageCached` events |
| `src/infrastructure/services/images/cache.rs` | Emit event tiap cache selesai |
| `Cargo.toml` | Tambah `reqwest`, `uuid`, `chrono` (jika belum ada) |
## Event Handlers (Subscribe)
Buat `src/subscribers/` untuk handler:
```rust
// src/subscribers/image_handler.rs
pub async fn handle_image_cached(event: CloudEvent) -> Result<()> {
// Log, notifikasi, update status
}
```
Daftarkan subscribers di `bootstrap/mod.rs` dengan spawn task:
```rust
tokio::spawn(async move {
let mut stream = dapr_client.subscribe("pubsub", "hub.image.cached");
while let Some(event) = stream.next().await {
handle_image_cached(event).await;
}
});
```
## Testing Event-Driven Code
```rust
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_publish_event() {
let client = MockDaprClient::new();
client.expect_publish()
.with(...)
.returning(|_| Ok(()));
// ... test
}
}
```
-96
View File
@@ -1,96 +0,0 @@
---
name: hub-rules
description: Aturan repository, arsitektur hub, submodule, dan workflow Asepharyana Hub
---
# Asepharyana Hub — Repository Rules
## Struktur Repository
```
asepharyana-hub/
├── apps/ # Git submodules — source code aplikasi
├── docs/ # Dokumentasi, ADR, deployment guide
├── infra/ # Infrastructure as code
│ ├── compose/ # Satu compose file per service
│ ├── dapr/ # Dapr component configs
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Static & dynamic Traefik config
├── scripts/ # Utility scripts (cleanup, update-deps)
└── .github/workflows/ # CI/CD pipelines
```
### Aturan Submodule
- Setiap aplikasi di `apps/` adalah **submodule** ke repo terpisah.
- Perubahan kode aplikasi dilakukan di **repo masing-masing**, bukan di sini.
- Submodule pointer diupdate oleh CI/CD (bukan manual).
## Infrastructure Patterns
### Networking
- Semua service join **`app-shared-net`** (external Docker bridge)
- Service discovery via Docker DNS (container alias)
- Traefik sebagai ingress untuk HTTP/S eksternal
- Tailscale untuk cross-VPS (PostgreSQL, Redis)
### Compose File Pattern
```yaml
services:
<service>:
container_name: <service>
image: ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<sha>
restart: always
networks:
app-shared-net:
aliases:
- <service>
env_file:
- ../../.env
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dapr Sidecar Pattern
```yaml
<service>-dapr:
container_name: <service>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
nats:
condition: service_started
dapr-placement:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<service>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--resources-path=/components'
volumes:
- ../../infra/dapr/components:/components
```
### Traefik Routing
- Router + service definition di `infra/traefik/dynamic/apps.yaml`
- Subdomain pattern: `<service>.asepharyana.my.id` + `<service>.asepharyana.web.id`
- TLS cert dari volume mount (bukan auto-acme)
### CI/CD
- `docker-build-push.yml` — build per service, push ke GHCR, update compose manifest
- `deploy-docker.yml` — SSH ke orangevps, pull images, restart
- Selective deploy: hanya compose file yg berubah
## Deployment Order
1. `shared.yml` (Redis)
2. `nats.yml` (NATS message bus)
3. `dapr.yml` (Dapr placement)
4. `traefik.yml` (Reverse proxy)
5. Service compose files (apps + Dapr sidecar)
-32
View File
@@ -1,32 +0,0 @@
**/.git
**/.gitmodules
**/node_modules
**/dist
**/.output
**/target
**/.svelte-kit
**/.next
**/.DS_Store
**/build
!apps/*/scripts/build/
!apps/*/src/**/build/
**/*.log
**/*.pem
.env
.env.*
!.env.example
# IDE and temporary files
**/.vscode
**/.idea
**/tmp
**/temp
**/.cache
**/coverage
**/.npm
**/.bun
**/.pnpm-store
**/.yarn
**/.cargo-ok
**/*.swp
**/*~
+17
View File
@@ -0,0 +1,17 @@
name: Dependabot Auto-Merge
on: pull_request
permissions:
contents: write
pull-requests: write
jobs:
auto-merge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- name: Enable auto-merge for Dependabot PR
run: gh pr merge --auto --merge "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+14 -17
View File
@@ -1,21 +1,18 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for more information:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
# https://containers.dev/guide/dependabot
version: 2
updates:
- package-ecosystem: 'devcontainers'
directory: '/'
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: weekly
- package-ecosystem: 'github-actions'
directory: '/'
schedule:
interval: weekly
interval: "daily"
open-pull-requests-limit: 10
groups:
github-actions:
patterns:
- '*'
production:
dependency-type: "production"
update-types:
- "minor"
- "patch"
development:
dependency-type: "development"
update-types:
- "minor"
- "patch"
-239
View File
@@ -1,239 +0,0 @@
name: Deploy Docker to VPS
on:
workflow_run:
workflows: ['Build and Push Docker Images']
types:
- completed
branches:
- main
push:
branches:
- main
paths:
- 'infra/**'
- '.github/workflows/deploy-docker.yml'
- '.github/workflows/docker-build-push.yml'
workflow_dispatch:
# Prevent multiple deployments from running simultaneously
concurrency:
group: deploy-vps
cancel-in-progress: false
permissions:
contents: read
packages: read
jobs:
deploy:
runs-on: ubuntu-latest
timeout-minutes: 30
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || github.event.workflow_run.conclusion == 'success'
steps:
- name: Checkout repository
uses: actions/checkout@v7
with:
fetch-depth: 1
submodules: false
- name: Deploy to VPS
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
VPS_TARGET_DIR: ${{ secrets.VPS_TARGET_DIR }}
ENV_FILE_PRODUCTION: ${{ secrets.ENV_FILE_PRODUCTION }}
GHCR_USERNAME: ${{ github.actor }}
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
echo "Deploy event: ${{ github.event_name }}"
echo "Deploy ref: ${{ github.ref }}"
echo "Deploy sha: ${{ github.sha }}"
if [ -z "${SSH_PRIVATE_KEY:-}" ] || [ -z "${VPS_HOST:-}" ] || [ -z "${VPS_USER:-}" ] || [ -z "${VPS_TARGET_DIR:-}" ]; then
echo "❌ Deployment secrets are not fully configured. Please set SSH_PRIVATE_KEY, VPS_HOST, VPS_USER, and VPS_TARGET_DIR."
exit 1
fi
mkdir -p ~/.ssh
echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -H -t ed25519,rsa "$VPS_HOST" >> ~/.ssh/known_hosts
# Use SSH multiplexing for faster subsequent commands
SSH_OPTS=(-o ControlMaster=auto -o ControlPath=/tmp/ssh-%r@%h:%p -o ControlPersist=600 -o StrictHostKeyChecking=yes)
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "mkdir -p $VPS_TARGET_DIR && mkdir -p $VPS_TARGET_DIR/infra/compose"
echo "$ENV_FILE_PRODUCTION" > .env.prod
scp "${SSH_OPTS[@]}" .env.prod "$VPS_USER@$VPS_HOST:$VPS_TARGET_DIR/.env"
echo "🔐 Logging in to GitHub Container Registry..."
printf '%s' "$GHCR_TOKEN" | ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "docker login ghcr.io -u '$GHCR_USERNAME' --password-stdin"
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "export VPS_TARGET_DIR=$VPS_TARGET_DIR; bash -s" <<'EOF'
set -euo pipefail
cd "$VPS_TARGET_DIR"
# Ensure shared network exists
docker network inspect app-shared-net >/dev/null 2>&1 || docker network create app-shared-net
echo "🔄 Synchronizing repository..."
if [ ! -d ".git" ]; then
echo "Initializing git repository..."
git init
git remote add origin https://github.com/asepharyana/asepharyana-hub.git
fi
git fetch origin main --depth=1 || true
# Detect changed files before resetting
ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/hub.yml infra/compose/tools.yml infra/compose/llm-api.yml infra/compose/nats.yml infra/compose/dapr.yml infra/compose/observability.yml"
TRAEFIK_DYNAMIC_DIR="infra/traefik/dynamic"
if git rev-parse HEAD >/dev/null 2>&1; then
BEFORE_REV=$(git rev-parse HEAD)
git reset --hard FETCH_HEAD
AFTER_REV=$(git rev-parse HEAD)
if [ "$BEFORE_REV" = "$AFTER_REV" ]; then
echo "ℹ️ No new commits detected. Using full file list for safety."
TARGET_COMPOSE=""
else
CHANGED=$(git diff --name-only "$BEFORE_REV" "$AFTER_REV" || true)
echo "📄 Changed files:"
echo "$CHANGED"
# Detect compose stack changes
CHANGED_COMPOSE=$(echo "$CHANGED" | grep '^infra/compose/.*\.yml$' || true)
TARGET_COMPOSE=""
for f in $CHANGED_COMPOSE; do
case " $ALL_COMPOSE_FILES " in
*" $f "*) TARGET_COMPOSE="$TARGET_COMPOSE $f" ;;
esac
done
TARGET_COMPOSE=$(printf '%s' "$TARGET_COMPOSE" | xargs || true)
if [ -n "$TARGET_COMPOSE" ]; then
echo "🎯 Detected compose stack changes in: $TARGET_COMPOSE"
else
echo "ℹ️ No stack compose files changed."
fi
# Detect Traefik dynamic config changes
CHANGED_TRAEFIK=$(echo "$CHANGED" | grep "^$TRAEFIK_DYNAMIC_DIR/" || true)
if [ -n "$CHANGED_TRAEFIK" ]; then
echo "🎯 Detected Traefik dynamic config changes:"
echo "$CHANGED_TRAEFIK"
RELOAD_TRAEFIK="true"
else
echo "ℹ️ No Traefik dynamic config changes."
fi
# Detect infra file changes (Dockerfiles, config, traefik static)
CHANGED_INFRA=$(echo "$CHANGED" | grep '^infra/' | grep -v '^infra/compose/' || true)
if [ -n "$CHANGED_INFRA" ]; then
echo "📦 Detected other infra file changes:"
echo "$CHANGED_INFRA"
fi
fi
else
git reset --hard FETCH_HEAD
TARGET_COMPOSE=""
fi
if command -v "docker" >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then
COMPOSE_CMD="docker compose"
elif command -v docker-compose >/dev/null 2>&1; then
COMPOSE_CMD="docker-compose"
else
echo "❌ docker compose is not installed on the remote host."
exit 1
fi
# Always include ALL compose files for dependency resolution
COMPOSE_ARGS=""
for f in $ALL_COMPOSE_FILES; do
if [ -f "$f" ]; then
COMPOSE_ARGS="$COMPOSE_ARGS -f $f"
fi
done
if [ -n "$TARGET_COMPOSE" ]; then
# Extract service names from target compose file(s) for selective up
TARGET_SERVICES=""
for f in $TARGET_COMPOSE; do
if [ -f "$f" ]; then
svcs=$($COMPOSE_CMD -f "$f" config --services 2>/dev/null | tr '\n' ' ' | xargs)
TARGET_SERVICES="$TARGET_SERVICES $svcs"
fi
done
TARGET_SERVICES=$(echo "$TARGET_SERVICES" | xargs) # trim whitespace
echo "🎯 Selective update for services: $TARGET_SERVICES"
else
echo "🚀 Performing full deployment of all services..."
TARGET_SERVICES=""
fi
echo "📥 Pulling images for target services..."
export DOCKER_CLI_EXPERIMENTAL=enabled
PULL_SUCCESS=false
# Retry pull up to 3 times to handle transient Docker attestation lease errors
for attempt in 1 2 3; do
echo "Pull attempt $attempt/3..."
if $COMPOSE_CMD $COMPOSE_ARGS --env-file .env pull $TARGET_SERVICES; then
echo "✅ Pull succeeded on attempt $attempt"
PULL_SUCCESS=true
break
else
echo "⚠️ Pull attempt $attempt failed. Retrying in 5s..."
sleep 5
fi
done
if [ "$PULL_SUCCESS" != "true" ]; then
echo "❌ Failed to pull images after 3 attempts."
exit 1
fi
echo "🧹 Clearing Git locks..."
rm -f .git/shallow.lock || true
echo "🧹 Removing stale target containers by container_name..."
# Extract all explicitly defined container_names from compose files and remove them to prevent conflicts
if [ -n "$TARGET_COMPOSE" ]; then
for f in $TARGET_COMPOSE; do
if [ -f "$f" ]; then
grep "container_name:" "$f" | awk '{print $2}' | while read -r cname; do
docker rm -f "$cname" >/dev/null 2>&1 || true
done
fi
done
else
for f in $ALL_COMPOSE_FILES; do
if [ -f "$f" ]; then
grep "container_name:" "$f" | awk '{print $2}' | while read -r cname; do
docker rm -f "$cname" >/dev/null 2>&1 || true
done
fi
done
fi
echo "🆙 Starting services..."
echo "🔍 Debug: Current docker containers:"
docker ps -a
if [ -n "$TARGET_SERVICES" ]; then
$COMPOSE_CMD $COMPOSE_ARGS --env-file .env up -d $TARGET_SERVICES
else
$COMPOSE_CMD $COMPOSE_ARGS --env-file .env up -d --remove-orphans
fi
# ── Traefik reload ──
if [ "${RELOAD_TRAEFIK:-false}" = "true" ]; then
echo "🔄 Traefik dynamic config changed — reloading Traefik..."
# Traefik watches the dynamic config dir (providers.file.watch=true),
# but send SIGHUP as insurance
docker kill --signal HUP traefik 2>/dev/null || docker exec traefik kill -HUP 1 2>/dev/null || true
echo "✅ Traefik reload signal sent"
fi
EOF
-338
View File
@@ -1,338 +0,0 @@
name: Build and Push Docker Images
on:
push:
branches:
- main
paths:
- 'apps/scraper/**'
- 'apps/hub/**'
- 'apps/tools/**'
- 'apps/llm-api/**'
- '.github/workflows/docker-build-push.yml'
- 'infra/**'
- '!infra/compose/**'
repository_dispatch:
types: [submodule-updated]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
REGISTRY: ghcr.io
IMAGE_NAME_PREFIX: asepharyana/asepharyana-hub
jobs:
# ──────────────────────────────────────────────
# Phase 1: Detect which services have changed
# ──────────────────────────────────────────────
changes:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
scraper-api: ${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }}
hub: ${{ steps.filter.outputs['hub'] == 'true' || steps.dispatch.outputs['hub'] == 'true' || github.event_name == 'workflow_dispatch' }}
tools: ${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}
llm-api: ${{ steps.filter.outputs['llm-api'] == 'true' || steps.dispatch.outputs['llm-api'] == 'true' || github.event_name == 'workflow_dispatch' }}
steps:
- uses: actions/checkout@v7
with:
submodules: false
fetch-depth: 2
- name: Detect changed services
id: filter
if: github.event_name == 'push'
env:
BEFORE: ${{ github.event.before }}
AFTER: ${{ github.sha }}
run: |
set -euo pipefail
if [ -z "${BEFORE:-}" ] || [[ "$BEFORE" =~ ^0+$ ]]; then
CHANGED_FILES=$(git ls-files)
else
git fetch --no-tags --depth=2 origin "$BEFORE" || true
CHANGED_FILES=$(git diff --name-only "$BEFORE" "$AFTER")
fi
changed() {
printf '%s\n' "$CHANGED_FILES" | grep -Eq "$1" && echo true || echo false
}
echo "scraper-api=$(changed '^(apps/scraper(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/scraper\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "hub=$(changed '^(apps/hub(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/hub\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "tools=$(changed '^(apps/tools(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/tools\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "llm-api=$(changed '^(apps/llm-api(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/llm-api\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
- name: Parse repository_dispatch payload
id: dispatch
if: github.event_name == 'repository_dispatch'
env:
SERVICE: ${{ github.event.client_payload.service }}
SHA: ${{ github.event.client_payload.sha }}
run: |
set -euo pipefail
if [ -z "${SERVICE:-}" ]; then
echo "::error::repository_dispatch payload missing service"
exit 1
fi
if [ -z "${SHA:-}" ]; then
echo "::error::repository_dispatch payload missing sha"
exit 1
fi
case "$SERVICE" in
scraper-api|hub|tools|llm-api) ;;
*)
exit 1
;;
esac
if ! [[ "$SHA" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "::error::Invalid sha '$SHA'. Expected 40 hex characters"; fi
SERVICES=(scraper-api hub tools llm-api)
for svc in "${SERVICES[@]}"; do
if [ "$SERVICE" = "$svc" ]; then
echo "${svc}=true" >> "$GITHUB_OUTPUT"
else
echo "${svc}=false" >> "$GITHUB_OUTPUT"
fi
done
- name: Set matrix
id: set-matrix
run: |
SERVICES=()
add_service() {
SERVICES+=("{\"id\":\"$1\",\"target\":\"$2\",\"path\":\"$3\"}")
}
if [ "${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "scraper-api" "docker-scraper" "apps/scraper"; fi
if [ "${{ steps.filter.outputs['hub'] == 'true' || steps.dispatch.outputs['hub'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "hub" "docker-hub" "apps/hub"; fi
if [ "${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "tools" "docker-tools" "apps/tools"; fi
if [ "${{ steps.filter.outputs['llm-api'] == 'true' || steps.dispatch.outputs['llm-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "llm-api" "docker-llm-api" "apps/llm-api"; fi
JSON_ARRAY="[$(IFS=,; echo "${SERVICES[*]}")]"
echo "matrix=$JSON_ARRAY" >> $GITHUB_OUTPUT
wait-submodule-ref:
needs: [changes]
if: github.event_name == 'repository_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Wait for submodule ref
env:
SERVICE: ${{ github.event.client_payload.service }}
SHA: ${{ github.event.client_payload.sha }}
run: |
set -euo pipefail
case "$SERVICE" in
"scraper-api") REPO="https://github.com/asepharyana/asepharyana-hub-scraper.git" ;;
"hub") REPO="https://github.com/asepharyana/asepharyana-hub-hub.git" ;;
"tools") echo "tools is built from monorepo, no submodule wait needed"; exit 0 ;;
"llm-api") REPO="https://github.com/asepharyana/asepharyana-hub-llm-api.git" ;;
*)
echo "::error::Unsupported service '$SERVICE'"
exit 1
;;
esac
echo "Waiting for $SERVICE commit $SHA in $REPO"
TMPDIR=$(mktemp -d)
git init "$TMPDIR/probe" >/dev/null
git -C "$TMPDIR/probe" remote add origin "$REPO"
for attempt in {1..30}; do
if git -C "$TMPDIR/probe" fetch --depth=1 origin "$SHA" >/dev/null 2>&1; then
echo "Submodule commit $SHA is fetchable for $SERVICE"
rm -rf "$TMPDIR"
exit 0
fi
echo "Attempt $attempt/30: $SHA not fetchable yet; waiting 10s"
sleep 10
done
rm -rf "$TMPDIR"
echo "::error::Submodule commit $SHA for $SERVICE was not fetchable after 300s"
exit 1
# ─────────────────────────────────────────────────
# Phase 2: Build and Push Images (Matrix)
# ─────────────────────────────────────────────────
build:
needs: [changes, wait-submodule-ref]
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.changes.outputs.matrix) }}
if: |
always() &&
needs.changes.result == 'success' &&
(needs.wait-submodule-ref.result == 'success' || needs.wait-submodule-ref.result == 'skipped') &&
needs.changes.outputs.matrix != '[]'
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v7
with:
submodules: false
- name: Sync submodule locally
env:
EVENT_NAME: ${{ github.event_name }}
DISPATCH_SHA: ${{ github.event.client_payload.sha }}
SUBMODULE_PATH: ${{ matrix.path }}
run: |
set -euo pipefail
git submodule update --init --recursive "$SUBMODULE_PATH"
if [ "$EVENT_NAME" = "repository_dispatch" ] && [ -n "${DISPATCH_SHA:-}" ]; then
cd "$SUBMODULE_PATH"
git fetch origin "$DISPATCH_SHA"
git checkout "$DISPATCH_SHA"
cd "${GITHUB_WORKSPACE}"
fi
- uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Docker metadata
id: meta
run: |
SVC_NAME="${{ matrix.id }}"
SHORT=${GITHUB_SHA:0:7}
echo "image=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}" >> $GITHUB_OUTPUT
echo "tags=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:sha-${SHORT}" >> $GITHUB_OUTPUT
echo "cache-registry=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:buildcache" >> $GITHUB_OUTPUT
case "$SVC_NAME" in
"scraper-api") echo "dockerfile=infra/docker/scraper.Dockerfile" >> $GITHUB_OUTPUT ;;
"hub") echo "dockerfile=infra/docker/hub.Dockerfile" >> $GITHUB_OUTPUT ;;
"tools") echo "dockerfile=infra/docker/tools.Dockerfile" >> $GITHUB_OUTPUT ;;
"llm-api") echo "dockerfile=infra/docker/llm-api.Dockerfile" >> $GITHUB_OUTPUT ;;
esac
- name: Build and Push Docker image
uses: docker/build-push-action@v7
with:
context: .
file: ${{ steps.meta.outputs.dockerfile }}
push: true
tags: ${{ steps.meta.outputs.tags }}
build-args: |
COMMIT_COUNT=${{ env.NR_COMMIT_COUNT || github.run_number }}
COMMIT_SHA=${{ env.NR_COMMIT_SHA || github.sha }}
cache-from: type=registry,ref=${{ steps.meta.outputs['cache-registry'] }}
cache-to: type=registry,ref=${{ steps.meta.outputs['cache-registry'] }},mode=max
# ──────────────────────────────────────────────
# Phase 3: Update Manifests and Submodule Refs
# ──────────────────────────────────────────────
update-manifest:
needs: [changes, wait-submodule-ref, build]
if: |
always() &&
needs.changes.result == 'success' &&
(needs.wait-submodule-ref.result == 'success' || needs.wait-submodule-ref.result == 'skipped') &&
(needs.build.result == 'success' || needs.build.result == 'skipped')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: write
steps:
- uses: actions/checkout@v7
with:
submodules: false
token: ${{ secrets.GITHUB_TOKEN }}
ref: main
- name: Update tags and submodules
run: |
SHORT_SHA=${GITHUB_SHA:0:7}
TAG="sha-$SHORT_SHA"
CHANGED=false
declare -A SERVICES
SERVICES["scraper-api"]="scraper.yml"
SERVICES["hub"]="hub.yml"
SERVICES["tools"]="tools.yml"
SERVICES["llm-api"]="llm-api.yml"
declare -A PATHS
PATHS["scraper-api"]="apps/scraper"
PATHS["hub"]="apps/hub"
PATHS["tools"]="apps/tools"
PATHS["llm-api"]="apps/llm-api"
# Use git config for possible commits
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
for id in "${!SERVICES[@]}"; do
SHOULD_HAVE_RUN=false
if [ "${{ needs.changes.outputs['scraper-api'] }}" == "true" ] && [ "$id" == "scraper-api" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['hub'] }}" == "true" ] && [ "$id" == "hub" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['tools'] }}" == "true" ] && [ "$id" == "tools" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['llm-api'] }}" == "true" ] && [ "$id" == "llm-api" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then SHOULD_HAVE_RUN=true; fi
if [ "$SHOULD_HAVE_RUN" == "true" ]; then
COMPOSE_FILE="infra/compose/${SERVICES[$id]}"
if [ -f "$COMPOSE_FILE" ]; then
echo "Updating $COMPOSE_FILE to $TAG"
sed -i "s|image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME_PREFIX }}/$id:.*|image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME_PREFIX }}/$id:$TAG|g" "$COMPOSE_FILE"
git add "$COMPOSE_FILE"
CHANGED=true
fi
# If it's a repository_dispatch for this specific service, update its submodule pointer
if [ "${{ github.event_name }}" == "repository_dispatch" ] && [ "${{ github.event.client_payload.service }}" == "$id" ]; then
SHA_DISPATCH="${{ github.event.client_payload.sha }}"
SUB_PATH="${PATHS[$id]}"
if [ -n "$SHA_DISPATCH" ]; then
echo "Updating submodule $SUB_PATH to $SHA_DISPATCH"
git submodule update --init "$SUB_PATH"
git -C "$SUB_PATH" fetch origin "$SHA_DISPATCH"
git -C "$SUB_PATH" checkout "$SHA_DISPATCH"
git add "$SUB_PATH"
CHANGED=true
fi
fi
fi
done
if [ "$CHANGED" == "true" ]; then
git commit -m "chore: update manifests and submodules [skip ci]"
for attempt in {1..3}; do
if git pull --rebase origin main && git push origin main; then
exit 0
fi
echo "Manifest push attempt $attempt/3 failed; retrying"
git rebase --abort || true
git pull --rebase origin main || true
sleep 5
done
echo "::error::Failed to push manifest update after 3 attempts"
exit 1
else
echo "No changes detected."
fi
+101
View File
@@ -0,0 +1,101 @@
name: Deploy Infra Config
on:
push:
branches: [main]
paths:
- 'infra/caddy/**'
- 'infra/firewall/**'
- 'infra/systemd/**'
- 'infra/prometheus/**'
workflow_dispatch:
concurrency:
group: infra-deploy
cancel-in-progress: false
permissions:
contents: read
env:
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
jobs:
deploy-infra:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7
- name: Validate Caddyfile syntax
run: |
curl -fsSL "https://caddyserver.com/api/download?os=linux&arch=amd64" -o /tmp/caddy
chmod +x /tmp/caddy
/tmp/caddy validate --config infra/caddy/Caddyfile.prod --adapter caddyfile 2>&1 | tail -5
echo "✅ Caddyfile valid"
- name: Setup SSH key
env:
SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
run: |
mkdir -p ~/.ssh
echo "$SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
sed -i 's/\r$//' ~/.ssh/id_ed25519
ssh-keygen -y -f ~/.ssh/id_ed25519 >/dev/null 2>&1 || { echo "SSH key invalid"; exit 1; }
ssh-keyscan -H "$VPS_HOST" >> ~/.ssh/known_hosts 2>/dev/null
- name: Sync Caddyfile to VPS
run: |
set -e
ssh "$VPS_USER@$VPS_HOST" "sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-previous"
scp -q infra/caddy/Caddyfile.prod "$VPS_USER@$VPS_HOST":/tmp/Caddyfile.new
ssh "$VPS_USER@$VPS_HOST" "sudo cp /tmp/Caddyfile.new /etc/caddy/Caddyfile && sudo rm -f /tmp/Caddyfile.new"
echo "✅ Caddyfile synced"
- name: Sync systemd drop-ins to VPS
run: |
set -e
if [ -d infra/systemd ]; then
for f in infra/systemd/*; do
[ -f "$f" ] || continue
base=$(basename "$f")
echo " syncing $base"
scp -q "$f" "$VPS_USER@$VPS_HOST":/tmp/"$base"
ssh "$VPS_USER@$VPS_HOST" "sudo mkdir -p /etc/systemd/system && sudo cp /tmp/$base /etc/systemd/system/$base && sudo rm -f /tmp/$base"
done
ssh "$VPS_USER@$VPS_HOST" "sudo systemctl daemon-reload"
echo "✅ systemd drop-ins synced"
else
echo "no infra/systemd/ files"
fi
- name: Sync prometheus targets to VPS
run: |
set -e
if [ -f infra/prometheus/targets.yml ]; then
scp -q infra/prometheus/targets.yml "$VPS_USER@$VPS_HOST":/tmp/targets.yml
ssh "$VPS_USER@$VPS_HOST" "sudo cp /tmp/targets.yml /etc/prometheus/targets.yml 2>/dev/null && sudo rm -f /tmp/targets.yml && sudo systemctl reload prometheus 2>/dev/null || true"
echo "✅ prometheus targets synced"
else
echo "no infra/prometheus/targets.yml"
fi
- name: Reload Caddy
run: |
ssh "$VPS_USER@$VPS_HOST" "sudo systemctl reload caddy || sudo systemctl restart caddy"
sleep 3
ssh "$VPS_USER@$VPS_HOST" "systemctl is-active caddy"
- name: Verify routes
run: |
set -e
for u in hub.asepharyana.my.id scraper.asepharyana.my.id tools.asepharyana.my.id wiki.asepharyana.my.id upload.asepharyana.my.id ai.asepharyana.my.id; do
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "https://$u/" || true)
echo "$u -> $code"
case "$code" in
000|502|503|504) echo "::error::$u bad status $code"; exit 1 ;;
esac
done
echo "✅ All routes reachable"
-28
View File
@@ -1,28 +0,0 @@
name: Lint
on:
workflow_dispatch:
pull_request:
branches: [main]
paths:
- 'biome.json'
- '*.json'
- '*.js'
push:
branches: [main]
paths:
- 'biome.json'
- '*.json'
- '*.js'
jobs:
biome:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- uses: oven-sh/setup-bun@v2
- run: bun install --frozen-lockfile
- run: bun run ci
-30
View File
@@ -1,30 +0,0 @@
name: Security
on:
pull_request:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Every Monday
jobs:
codeql:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
security-events: write
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 2
submodules: recursive
- uses: github/codeql-action/init@v4
with:
languages: rust
- name: Build Rust projects for CodeQL analysis
run: |
cargo build --manifest-path apps/scraper/Cargo.toml
cargo build --manifest-path apps/llm-api/Cargo.toml
- uses: github/codeql-action/analyze@v4
-86
View File
@@ -1,86 +0,0 @@
name: Update Submodule Pointer
on:
repository_dispatch:
types: [submodule-updated]
permissions:
contents: write
jobs:
update:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Validate payload
env:
SERVICE: ${{ github.event.client_payload.service }}
SHA: ${{ github.event.client_payload.sha }}
run: |
set -euo pipefail
if [ -z "${SERVICE:-}" ]; then
echo "::error::Missing service in payload"
exit 1
fi
if [ -z "${SHA:-}" ]; then
echo "::error::Missing sha in payload"
exit 1
fi
if ! [[ "$SHA" =~ ^[0-9a-fA-F]{40}$ ]]; then
echo "::error::Invalid sha '$SHA'. Expected 40 hex characters."
exit 1
fi
case "$SERVICE" in
scraper-api|hub|llm-api|tools) ;;
*)
echo "::error::Unsupported service '$SERVICE'"
exit 1
;;
esac
echo "Payload validated: $SERVICE → $SHA"
- name: Update submodule pointer
env:
SERVICE: ${{ github.event.client_payload.service }}
SHA: ${{ github.event.client_payload.sha }}
run: |
set -euo pipefail
# Map service name to submodule path
case "$SERVICE" in
scraper-api) SUBMODULE_PATH="apps/scraper" ;;
llm-api) SUBMODULE_PATH="apps/llm-api" ;;
*) SUBMODULE_PATH="apps/${SERVICE}" ;;
esac
echo "Updating ${SUBMODULE_PATH} to ${SHA}"
git submodule update --init "${SUBMODULE_PATH}"
cd "${SUBMODULE_PATH}"
git fetch --depth=1 origin master 2>/dev/null || git fetch --depth=1 origin main
git checkout "${SHA}"
cd "${GITHUB_WORKSPACE}"
git add "${SUBMODULE_PATH}"
git diff --cached --quiet && exit 0
git config user.name "monrepo-bot"
git config user.email "monrepo-bot@users.noreply.github.com"
git commit -m "chore: update ${SERVICE} to ${SHA:0:12}"
for attempt in {1..3}; do
if git pull --rebase origin main && git push origin main; then
echo "✅ Push succeeded on attempt $attempt"
exit 0
fi
echo "⚠️ Push attempt $attempt/3 failed; retrying..."
git rebase --abort 2>/dev/null || true
sleep 3
done
echo "::error::Failed to push submodule update after 3 attempts"
exit 1
+2
View File
@@ -11,6 +11,7 @@ node_modules
.turbo/
# IDEs and editors
/.idea
.serena/
.project
.classpath
.c9/
@@ -63,3 +64,4 @@ docs/todo.md
**/vendor/
.codegraph/
result
-15
View File
@@ -1,15 +0,0 @@
[submodule "apps/scraper"]
path = apps/scraper
url = https://github.com/asepharyana/asepharyana-hub-scraper.git
[submodule "apps/hub"]
path = apps/hub
url = https://github.com/asepharyana/asepharyana-hub-hub.git
[submodule "apps/tools"]
path = apps/tools
url = https://github.com/asepharyana/asepharyana-hub-tools.git
[submodule "plugins/hub-guide"]
path = plugins/hub-guide
url = https://github.com/asepharyana/asepharyana-hub-guide.git
[submodule "apps/llm-api"]
path = apps/llm-api
url = https://github.com/asepharyana/asepharyana-hub-llm-api.git
-65
View File
@@ -1,65 +0,0 @@
---
name: commit-convention
description: Commit message convention — type(scope): description for Asepharyana Hub
---
# Commit Convention — Asepharyana Hub
## Format
```
<type>(<scope>): <description>
[optional body]
[optional footer]
```
## Types
| Type | Usage |
| ---------- | ------------------------------------ |
| `feat` | Fitur baru |
| `fix` | Bug fix |
| `chore` | Maintenance, config, tooling |
| `docs` | Dokumentasi |
| `refactor` | Perubahan kode tanpa fungsional baru |
| `test` | Nambah/update test |
| `ci` | CI/CD workflows |
| `perf` | Optimasi performa |
| `style` | Formatting (tanda kutip, dll) |
## Scopes
| Scope | Area |
| ------------- | --------------------------------- |
| `scraper` | apps/scraper submodule |
| `infra` | infra/ (compose, traefik, docker) |
| `ci` | .github/workflows/ |
| `dapr` | Dapr config & sidecar |
| `nats` | NATS message bus |
| `docs` | Dokumentasi |
| `deps` | Dependencies |
| `scripts` | Utility scripts |
| `root` | Root config files |
## Contoh
```
feat(scraper): add anime detail caching via Dapr pubsub
fix(infra): correct NATS CLI flags for JetStream
chore(deps): update biome to v2.5.3
docs(infra): add deployment order for Dapr services
ci(deploy): add nats.yml to ALL_COMPOSE_FILES
refactor(scraper): migrate EventBus from tokio broadcast to Dapr pubsub
```
## Aturan
1. **Wajib** menyertakan scope dalam tanda kurung
2. **Wajib** `Co-Authored-By` untuk commit yang digenerate AI
3. **Gunakan imperative mood**: "add" bukan "added" / "adds"
4. **Jangan capitalize** type: `feat:` bukan `Feat:`
5. **No period** di akhir subject baris
6. Body explain **why** dan **what**, bukan **how**
7. Refer issue dengan `Closes #123` atau `Fixes #123` di footer
-97
View File
@@ -1,97 +0,0 @@
---
name: deploy-workflow
description: CI/CD pipeline, Docker build patterns, manual deploy steps, and troubleshooting for Asepharyana Hub
---
# Deploy & Workflow — Asepharyana Hub
## CI/CD Pipeline
### Build Pipeline (`docker-build-push.yml`)
Trigger: push ke `main` yang touch `apps/**`, `infra/**`, `infra/docker/**`
1. **changes** — detect service mana yg berubah via git diff
2. **wait-submodule-ref** — (repository_dispatch only) tunggu SHA commit fetchable
3. **build** — matrix build per service, push ke GHCR (`sha-<short>` + `latest`)
4. **update-manifest** — update image tag di compose file, commit + push
### Deploy Pipeline (`deploy-docker.yml`)
Trigger: build selesai, atau push ke `main` touch `infra/**`
1. SSH ke `orangevps` (via `secrets.VPS_HOST`)
2. Sync repo (`git fetch --depth=1 + reset`)
3. Login ke GHCR
4. Deteksi compose file yg berubah
5. Pull images + restart container selektif
### Secrets Required
| Secret | Untuk |
|--------|-------|
| `SSH_PRIVATE_KEY` | SSH ke VPS |
| `VPS_HOST` | IP/host VPS (tailscale IP) |
| `VPS_USER` | SSH user, biasanya `root` |
| `VPS_TARGET_DIR` | Lokasi repo di VPS |
| `ENV_FILE_PRODUCTION` | .env content untuk production |
### Selective Deployment
- Hanya compose file yg berubah yang di-redeploy
- Selective: `UP_FLAGS="-d"` (tanpa `--remove-orphans`)
- Full deploy: `UP_FLAGS="-d --remove-orphans"`
## Docker Patterns
### Build dengan cargo-chef (Rust)
```dockerfile
FROM lukemathwalker/cargo-chef:latest-rust-1.89.0 AS chef
WORKDIR /app
FROM chef AS planner
COPY apps/scraper .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY apps/scraper .
RUN cargo build --release
```
### Runtime minimal untuk Rust binary
```dockerfile
FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3 && rm -rf /var/lib/apt/lists/*
```
## Manual Deploy Steps
```bash
# 1. Login GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u asepharyana --password-stdin
# 2. Full stack
docker compose -f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env up -d --remove-orphans
# 3. Selective (hanya satu service)
docker compose -f infra/compose/scraper.yml --env-file .env up -d
```
## Troubleshooting
### Container reach Tailscale
Pastikan route ke Tailscale di main table:
```bash
ip route add 100.64.0.0/10 dev tailscale0 table main
systemctl restart tailscale-routes
```
### Healthcheck gagal di scratch images
NATS dan Dapr placement pake scratch — tidak bisa healthcheck. Cukup `service_started` di depends_on.
### Dapr sidecar crash
```bash
docker logs scraper-api-dapr | grep -iE "fatal|error"
```
Penyebab umum: komponen config salah, NATS/Dapr placement belum siap.
-133
View File
@@ -1,133 +0,0 @@
---
name: event-driven
description: Event-driven architecture patterns with Dapr + NATS for Asepharyana Hub services
---
# Event-Driven Architecture — Asepharyana Hub
## Stack
- **Message Backbone**: NATS + JetStream (untuk streaming & job queue)
- **Pub/Sub Runtime**: Dapr sidecar per service (pubsub via Redis built-in)
- **State Store**: Dapr → Redis
## Event Topics Convention
```
hub.<domain>.<action>
Contoh:
hub.image.cached → Image selesai di-cache ke CDN
hub.image.repaired → Image diperbaiki (CNAME change)
hub.scrape.anime.done → Scrape anime selesai
hub.system.alert → Error/alert dari service
```
## CloudEvents Format
```json
{
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"subject": "anime-poster",
"id": "uuid-v4",
"time": "2026-07-21T10:00:00Z",
"datacontenttype": "application/json",
"data": { ... }
}
```
## Publish Event (Rust via HTTP API)
Gunakan `reqwest` langsung ke Dapr sidecar (SDK Rust masih experimental):
```rust
let event = serde_json::json!({
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"id": Uuid::new_v4().to_string(),
"time": chrono::Utc::now().to_rfc3339(),
"datacontenttype": "application/json",
"data": { "original_url": url, "cdn_url": cdn_url }
});
reqwest::Client::new()
.post("http://localhost:3500/v1.0/publish/pubsub/hub.image.cached")
.json(&event)
.send()
.await?;
```
## Service Invocation
```bash
curl http://localhost:3500/v1.0/invoke/<app-id>/method/<path>
```
## State Store
```bash
# Set
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
# Get
curl http://localhost:3500/v1.0/state/statestore/mykey
# Delete
curl -X DELETE http://localhost:3500/v1.0/state/statestore/mykey
```
## Scraper Event Integration
File yang perlu dimodifikasi untuk event-driven:
| File | Perubahan |
|------|-----------|
| `src/events/bus.rs` | Ganti backend dari tokio broadcast ke Dapr pub/sub |
| `src/bootstrap/mod.rs` | Init DaprClient, inject ke AppState |
| `src/presentation/state.rs` | Tambah `dapr_client` field |
| `src/proxy/use_cases.rs` | Publish `ImageRepaired` & `ImageCached` events |
| `src/infrastructure/services/images/cache.rs` | Emit event tiap cache selesai |
| `Cargo.toml` | Tambah `reqwest`, `uuid`, `chrono` (jika belum ada) |
## Event Handlers (Subscribe)
Buat `src/subscribers/` untuk handler:
```rust
// src/subscribers/image_handler.rs
pub async fn handle_image_cached(event: CloudEvent) -> Result<()> {
// Log, notifikasi, update status
}
```
Daftarkan subscribers di `bootstrap/mod.rs` dengan spawn task:
```rust
tokio::spawn(async move {
let mut stream = dapr_client.subscribe("pubsub", "hub.image.cached");
while let Some(event) = stream.next().await {
handle_image_cached(event).await;
}
});
```
## Testing Event-Driven Code
```rust
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_publish_event() {
let client = MockDaprClient::new();
client.expect_publish()
.with(...)
.returning(|_| Ok(()));
// ... test
}
}
```
-101
View File
@@ -1,101 +0,0 @@
---
name: hub-rules
description: Repository structure, submodule strategy, infrastructure patterns, and architecture of Asepharyana Hub
---
# Asepharyana Hub — Repository Rules
## Struktur Repository
```
asepharyana-hub/
├── apps/ # Git submodules — source code aplikasi
├── docs/ # Dokumentasi, ADR, deployment guide
├── infra/ # Infrastructure as code
│ ├── compose/ # Satu compose file per service
│ ├── dapr/ # Dapr component configs
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Static & dynamic Traefik config
├── scripts/ # Utility scripts (cleanup, update-deps)
└── .github/workflows/ # CI/CD pipelines
```
### Aturan Submodule
- Setiap aplikasi di `apps/` adalah **submodule** ke repo terpisah.
- Perubahan kode aplikasi dilakukan di **repo masing-masing**, bukan di sini.
- Submodule pointer diupdate oleh CI/CD (bukan manual).
## Infrastructure Patterns
### Networking
- Semua service join **`app-shared-net`** (external Docker bridge)
- Service discovery via Docker DNS (container alias)
- Traefik sebagai ingress untuk HTTP/S eksternal
- Tailscale untuk cross-VPS (PostgreSQL, Redis)
### Compose File Pattern
```yaml
services:
<service>:
container_name: <service>
image: ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<sha>
restart: always
networks:
app-shared-net:
aliases:
- <service>
env_file:
- ../../.env
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dapr Sidecar Pattern
```yaml
<service>-dapr:
container_name: <service>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
nats:
condition: service_started
dapr-placement:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<service>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--resources-path=/components'
volumes:
- ../../infra/dapr/components:/components
```
### Traefik Routing
- Router + service definition di `infra/traefik/dynamic/apps.yaml`
- Subdomain pattern: `<service>.asepharyana.my.id` + `<service>.asepharya.web.id`
- TLS cert dari volume mount (bukan auto-acme)
### Image Tagging
- `sha-<short-sha>` — immutable, untuk rollback
- `latest` — mutable, untuk convenience
- Registry: `ghcr.io/asepharyana/asepharyana-hub/<service>`
### CI/CD
- `docker-build-push.yml` — build per service, push ke GHCR, update compose manifest
- `deploy-docker.yml` — SSH ke orangevps, pull images, restart
- Selective deploy: hanya compose file yg berubah
## Deployment Order
1. `shared.yml` (Redis)
2. `nats.yml` (NATS message bus)
3. `dapr.yml` (Dapr placement)
4. `traefik.yml` (Reverse proxy)
5. Service compose files (apps + Dapr sidecar)
-1
View File
@@ -1 +0,0 @@
22.11.0
-2
View File
@@ -1,2 +0,0 @@
engine-strict=true
save-exact=true
-314
View File
@@ -1,314 +0,0 @@
# Architecture
## Hub Repository Structure Overview
```
asepharyana-hub/
├── apps/ # Application services (Git submodules)
│ └── scraper/ # Web scraper service
├── docs/ # Documentation
│ ├── adr/ # Architecture Decision Records
│ ├── add-new-app.md # Guide for adding new services
│ └── superpowers/ # Project capabilities tracking
├── infra/ # Infrastructure as code
│ ├── compose/ # Docker Compose files per service
│ ├── config/ # Infrastructure configuration
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Traefik reverse proxy config
│ └── dynamic/ # Dynamic routing rules (YAML)
├── scripts/ # Utility scripts
│ ├── git-hooks/ # Git hook scripts
│ ├── cleanup-ghcr.sh # GHCR image cleanup
│ └── update-deps.sh # Dependency update helper
├── .github/workflows/ # CI/CD pipelines
├── eslint.config.mjs # Root ESLint config
├── package.json # Root formatting/lint helper scripts
└── .prettierrc # Prettier formatting rules
```
## Technology Stack
### Services
| Service | Language/Runtime | Framework | Database | Key Libraries |
| --------- | ---------------- | --------- | -------- | ------------- |
| **scraper** | _(submodule)_ | — | — | — |
### Infrastructure
| Component | Technology | Purpose |
| ------------------ | ----------------------- | ---------------------------------------------------------------- |
| Reverse Proxy | Traefik v3.6 | TLS termination, routing, middleware (rate-limit, headers, auth) |
| Container Runtime | Docker + Docker Compose | Service isolation and orchestration |
| Container Registry | GHCR (ghcr.io) | Docker image storage |
| Networking | Tailscale | Secure overlay network between VPS nodes |
| Message Bus | NATS + JetStream | Event-driven pub/sub, job queues, streaming |
| Runtime Sidecar | Dapr | Service invocation, pub/sub abstraction, state management |
| Cache & State | Redis (Alpine) | Session store, rate limit counters, caching, Dapr state store |
| CI/CD | GitHub Actions | Build, test, deploy automation |
## Infrastructure
### Traefik Reverse Proxy
Traefik runs as the entry point for all HTTP/S traffic. It is configured via:
- **Static config**: CLI arguments in `infra/compose/traefik.yml` — entry points, providers, plugins
- **Dynamic config**: `infra/traefik/dynamic/` — routers, services, middlewares, TLS
- **Docker provider**: Auto-discovers containers with `traefik.enable=true` labels
- **File provider**: Loads `apps.yaml` (routers/services), `middlewares.yaml`, `ssl.yaml`
Key middleware chains (`infra/traefik/dynamic/middlewares.yaml`):
- `secure-headers` — SSL redirect, HSTS, XSS protection, CSP
- `compress` — Gzip compression for responses over 256 bytes
- `rate-limit` — 100 avg / 50 burst requests
- `buffer` — 10MB request/response body limit
- `block-sensitive-paths` — blocks `.env`, `.git`, `/wp-admin` etc.
- `common-chain` — composes secure-headers + compress + retry + rate-limit + buffer
All services route through Traefik on port 443 (TLS), with automatic HTTP-to-HTTPS redirect.
### Docker Compose
Each service has its own Compose file under `infra/compose/`. All services join the `app-shared-net` external Docker network, enabling inter-service communication by container name.
Shared services:
- `infra/compose/shared.yml` — Redis (alias: `redis`)
- `infra/compose/traefik.yml` — Traefik reverse proxy
Service compose files are combined during deployment:
```bash
docker compose -f traefik.yml -f shared.yml -f scraper.yml up -d
```
### Tailscale Networking
```mermaid
graph TB
subgraph "Tailnet (100.64.0.0/10)"
IMRNES["imrnes (100.121.180.82)"]
ORANGEVPS["orangevps (100.79.111.61)"]
ARCH["archlinux (100.84.39.83)"]
end
subgraph "imrnes Services"
PG[(PostgreSQL)]
REDIS[Redis]
end
subgraph "orangevps Containers"
TRAEFIK[Traefik :443]
SCRAPER[scraper-api :4091]
end
TRAEFIK --> SCRAPER
style IMRNES fill:#3a7,color:#fff
style ORANGEVPS fill:#37a,color:#fff
style ARCH fill:#773,color:#fff
```
Container-to-Tailscale connectivity requires a systemd service that adds a route to the main routing table:
```
ip route add 100.64.0.0/10 dev tailscale0 table main
```
This is managed by `/etc/systemd/system/tailscale-routes.service` on the `orangevps` VPS.
## Data Flow
### Request Flow (Production)
```mermaid
sequenceDiagram
participant User as Browser/Client
participant DNS as Cloudflare DNS
participant Traefik as Traefik Proxy
participant App as Application Container
participant DB as PostgreSQL (imrnes via Tailscale)
participant Redis as Redis (imrnes via Tailscale)
User->>DNS: asepharyana.my.id
DNS->>User: A/AAAA record → orangevps VPS IP
User->>Traefik: HTTPS request :443
Traefik->>Traefik: TLS termination
Traefik->>Traefik: Middleware chain (headers, rate-limit, buffer)
Traefik->>App: HTTP reverse-proxy (internal network)
alt Database query
App->>DB: sqlx/Drizzle query via Tailscale
DB-->>App: Result set
else Cache lookup
App->>Cache: GET/SET via Tailscale
Cache-->>App: Cached value
end
App-->>Traefik: HTTP response
Traefik-->>User: HTTPS response
```
### CI/CD Pipeline
```mermaid
flowchart LR
A[Push to main] --> B{Changed paths?}
B -->|apps/** or infra/docker/**| C[Build Docker Images]
B -->|infra/compose/**| D[Deploy to VPS]
B -->|apps/*/src/**/*.ts| E[Lint + TypeCheck]
C --> F[Push to GHCR]
F --> G[Update Compose tags]
G --> D
D --> H[SSH into VPS]
H --> I[Pull images]
I --> J[docker compose up -d]
subgraph "Build Phase"
C
F
G
end
subgraph "Deploy Phase"
D
H
I
J
end
```
## Deployment Architecture
### Image Tags
- Every push to `main` triggers Docker builds for changed services
- Images are tagged with both `latest` and `sha-<short-sha>` (e.g., `sha-b0ef947`)
- Compose files are auto-updated to pin the new SHA tag
- This enables deterministic rollbacks by reverting the compose file change
### VPS Deployment
The `orangevps` VPS (Tailscale `100.79.111.61`) hosts all application containers:
1. GitHub Actions SSHes into the VPS
2. Production secrets are written as `.env`
3. The repo is synchronized via `git pull`
4. Changed compose files are detected by `git diff`
5. Docker images are pulled (with retry logic for transient failures)
6. Old containers are removed by `container_name`
7. `docker compose up -d` brings up the new containers
8. Traefik automatically detects the new containers via Docker provider
### Selective Deployment
The deploy workflow supports selective updates — if only one compose file changed, only the corresponding service is pulled and recreated, avoiding disruption to other services.
```mermaid
graph TB
subgraph "Orange VPS"
DIR[/root/asepharyana-hub/]
ENV[.env]
COMPOSE[infra/compose/*.yml]
NET[app-shared-net]
DIR -->|git pull| COMPOSE
ENV -->|docker compose --env-file| COMPOSE
COMPOSE -->|docker compose pull| IMAGES[(GHCR Images)]
COMPOSE -->|docker compose up -d| CONT[Containers]
CONT --> NET
end
subgraph "GitHub Actions"
BUILD[Build & Push]
DEPLOY[Deploy Workflow]
BUILD -->|trigger| DEPLOY
DEPLOY -->|SSH| DIR
end
IMAGES -->|registry| GHCR[ghcr.io/asepharyana]
```
## Submodule Strategy
Each application lives in its own Git repository and is imported as a submodule into `apps/`. This approach:
- **Enables independent development** — each service can be developed, tested, and versioned separately
- **Pins exact commits** — the super-repository tracks exact submodule SHAs, enabling reproducible deployments
- **Supports `repository_dispatch`** — when a submodule receives a push, it can trigger the super-repository to build and deploy only that service
### Submodule Lifecycle
1. Developer pushes to a submodule (e.g., `apps/scraper`)
2. Submodule's GitHub Action dispatches `repository_dispatch` to the super-repo with the service name and new SHA
3. Super-repo detects the dispatch, waits for the SHA to be fetchable, then builds only that service
4. The compose manifest is updated and committed with the new SHA tag
5. The deploy workflow runs and updates only the changed containers
### Updating Submodules
```bash
# Update a single submodule to latest
cd apps/scraper
git checkout main
git pull
cd ../..
git add apps/scraper
git commit -m "chore(scraper): update submodule to latest"
```
## Service Mesh & Inter-Service Communication
### HTTP (External + Internal via Traefik)
External traffic and internal HTTP calls route through Traefik. Services on `app-shared-net` can also communicate directly by container name.
### Event-Driven (NATS + Dapr)
NATS with JetStream provides a persistent message backbone. Each service has a Dapr sidecar that abstracts pub/sub, service invocation, and state management.
```mermaid
graph TB
subgraph "External"
WWW[Internet]
end
subgraph "Orange VPS"
TRAEFIK[Traefik :443]
subgraph "app-shared-net"
NATS[NATS + JetStream<br/>:4222]
DAPR_PLACEMENT[Dapr Placement<br/>:50005]
subgraph "Service: scraper-api"
SCRAPER[scraper-api<br/>:4091]
DAPR_SIDECAR[Dapr Sidecar<br/>:3500]
SCRAPER --- DAPR_SIDECAR
end
end
DAPR_SIDECAR -.->|gRPC pub/sub| NATS
DAPR_SIDECAR -.->|placement| DAPR_PLACEMENT
end
WWW -->|HTTPS| TRAEFIK
TRAEFIK --> SCRAPER
```
### Communication Patterns
| Pattern | Mechanism | Use Case |
|---------|-----------|----------|
| External HTTP | Traefik → Service | User requests, API calls |
| Internal HTTP | Service → Service (via Traefik or direct) | Synchronous queries |
| Pub/Sub Event | Dapr sidecar → NATS JetStream | Async notifications, image cache events |
| Service Invocation | Dapr sidecar gRPC | Cross-service RPC with retry & observability |
| State Store | Dapr → Redis | Shared state, job progress |
## Observability
- **Traefik access logs**: JSON format, logged at INFO level
- **Traefik access logs**: JSON format, logged at INFO level
- **Dashboard**: Traefik dashboard at `traefik.asepharyana.my.id` (secured)
+10
View File
@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [2026-08-02]
### Changed
- **Infra overhaul**: Docker + Traefik dihapus dari produksi → Caddy 2.11.4 (reverse proxy, auto-TLS LE, HTTP/3) + Nix/systemd services.
- **Port migration**: semua service pindah ke port 4000-an (hub 4003, tools 4007/4008, scraper 4091, llm-api 4010, dll).
- **DB via PgBouncer pool**: semua service konek ke imrnes 100.121.180.82:6432 (bukan :5432 langsung).
- **Secrets**: Bitwarden Secrets Manager (BWS) sebagai central secret store, wrapper bws-exec.
- **Flake**: dibatasi x86_64-linux (nixpkgs 26.11 drop darwin).
## [Unreleased]
### Changed
+18 -19
View File
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Repository Overview
Asepharyana Hub is a **hub monorepo** for Asep Haryana Saputra's portfolio ecosystem. Application services live in separate repos imported as Git submodules under `apps/`. Infrastructure (Docker Compose, Traefik, Dapr) lives in `infra/`.
Asepharyana Hub is a **hub monorepo** for Asep Haryana Saputra's portfolio ecosystem. Application services live in separate repos imported as Git submodules under `apps/`. Production infrastructure: Caddy reverse proxy + Nix/systemd services (Docker/Traefik removed 2026-08-02; legacy configs under `infra/` marked LEGACY).
```
asepharyana-hub/
@@ -13,9 +13,9 @@ asepharyana-hub/
│ └── scraper/ # Rust scraper API (asepharyana-hub-scraper)
├── docs/ # ADRs, deployment guide, new-app guide
├── infra/
│ ├── compose/ # One Docker Compose file per service
│ ├── compose/ # Docker Compose files (LEGACY — Docker dihapus)
│ ├── dapr/ # Dapr config + component definitions
│ ├── docker/ # Dockerfiles per service
│ ├── docker/ # Dockerfiles (LEGACY)
│ └── traefik/ # Reverse proxy config (static + dynamic)
├── scripts/ # Utility scripts (cleanup, update-deps, git hooks)
└── .github/workflows/ # CI/CD pipelines
@@ -31,23 +31,22 @@ asepharyana-hub/
- `apps/tools` → `asepharyana/asepharyana-hub-tools`.
### Infrastructure Stack
- **Traefik v3.6** — reverse proxy, TLS termination, middleware chain, Prometheus metrics (`--metrics.prometheus=true`)
- **Caddy 2.11.4** — reverse proxy, TLS termination (auto-LE), HTTP/3, zstd/gzip, keep-alive tuning (`/etc/caddy/Caddyfile`, ref `infra/caddy/Caddyfile.prod`)
- **NATS + JetStream** — message broker with persistent streaming
- **Dapr** — sidecar runtime (pub/sub abstraction, state management, service invocation)
- **Redis (Alpine)** — cache, session store, Dapr state store & pub/sub backend
- **Prometheus** — metrics backend with Docker service discovery (`docker_sd_configs`). Auto-discovers containers with `prometheus.io/scrape=true` label.
- **Prometheus** — metrics backend with `file_sd_configs` target files.
- **Jaeger** — distributed tracing backend (all-in-one), OTLP receiver
- **Tailscale** — secure overlay network between VPS nodes (PostgreSQL on `imrnes`, containers on `orangevps`)
### Monitoring
- **Hub dashboard** at `/dashboard` (Next.js client page, auto-refresh 15s)
- **Dashboard API** at `/api/dashboard` — returns JSON with Docker containers, Jaeger traces, Prometheus metrics (RPS, latency, errors, node CPU/RAM/Disk)
- **Docker socket** mounted on `hub` container (`--group-add 988`) for container discovery
- **Prometheus** auto-scrapes Traefik for per-service request metrics
- **Dashboard API** at `/api/dashboard` — returns JSON with systemd services, Jaeger traces, Prometheus metrics (RPS, latency, errors, node CPU/RAM/Disk)
- **Prometheus** scrapes node-exporter + app metrics endpoints
### Networking
- All containers join `app-shared-net` (external Docker bridge network). Service discovery via Docker DNS (container name aliases).
- Traefik handles all external HTTP/S traffic on port 443.
- All services run as Nix/systemd units; inter-service via 127.0.0.1:<port>.
- Caddy handles all external HTTP/S traffic on port 443 (and HTTP/3 UDP).
- Cross-VPS traffic (DB, Redis) goes through Tailscale (`100.64.0.0/10`). Container-to-Tailscale connectivity requires a route in the main routing table (managed by `tailscale-routes.service`).
## Commands
@@ -62,7 +61,7 @@ bun run ci # Biome CI mode (no writes, exit code on issues)
bun run format # Format only
bun run lint # Lint only
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest . # Build image
# Nix build (produksi): nix build .#default --impure --option sandbox false
```
### Validate YAML
@@ -76,8 +75,8 @@ for f in infra/compose/*.yml; do docker compose -f "$f" config >/dev/null && ech
| Workflow | Trigger | Action |
|----------|---------|--------|
| `lint.yml` | PR/push to main touching `*.json`, `*.js`, `biome.json` | `bun run ci` (Biome lint) |
| `docker-build-push.yml` | Push to main touching `apps/**`/`infra/**`, or `repository_dispatch` | Build Docker images per changed service, push to GHCR, update compose manifests |
| `deploy-docker.yml` | After build completes, or push touching `infra/**` | SSH to VPS (orangevps), pull images, restart containers selectively |
| `deploy.yml` | Push to main | nix build → nix copy ssh:// → systemctl restart |
| `docker-build-push.yml` | LEGACY (Docker dihapus) | LEGACY |
| `security.yml` | PR to main + weekly Monday | CodeQL analysis (Rust) |
| `update-submodule.yml` | `repository_dispatch` | Update submodule pointer in hub repo |
@@ -99,11 +98,11 @@ Each service gets one compose file. Containers join `app-shared-net` with a `con
### Dapr Sidecar Pattern
Each app gets a companion `daprd` sidecar container. Dapr components (pubsub, statestore) are mounted from `infra/dapr/components/`. The sidecar communicates with NATS for pub/sub and Dapr placement for actor coordination.
### Traefik Routing
- Routers + services defined in `infra/traefik/dynamic/apps.yaml`
### Caddy Routing
- Site blocks in `/etc/caddy/Caddyfile` (ref `infra/caddy/Caddyfile.prod`)
- Subdomain pattern: `<service>.asepharyana.my.id` and `<service>.asepharya.web.id`
- TLS certs from volume mounts (not auto-ACME)
- Middleware chain: `secure-headers` → `compress` → `retry` → `rate-limit` → `buffer`
- Auto-TLS via Let's Encrypt
- Shared handler snippet `(proxy)`: `encode zstd gzip` + security headers + keep-alive tuning
### Image Tagging
- `sha-<short-sha>` — immutable, for deterministic rollbacks
@@ -115,9 +114,9 @@ Each app gets a companion `daprd` sidecar container. Dapr components (pubsub, st
1. Create a separate repo for the app code
2. Add as submodule: `git submodule add <url> apps/<name>`
3. Create Dockerfile in `infra/docker/`
3. Create Nix flake package + systemd unit
4. Create compose file in `infra/compose/` (app + Dapr sidecar)
5. Add Traefik router in `infra/traefik/dynamic/apps.yaml`
5. Add Caddy site block in `/etc/caddy/Caddyfile`
6. Add build job in `.github/workflows/docker-build-push.yml`
7. See `docs/add-new-app.md` for full guide
-22
View File
@@ -1,22 +0,0 @@
.PHONY: help dev update-submodules deploy init-submodules status
SHELL := /bin/bash
help: ## Show this help
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
dev: ## Start development infrastructure (Redis etc.)
docker compose -f infra/compose/shared.yml up -d
update-submodules: ## Update all git submodules to latest remote
git submodule update --remote --merge --recursive
deploy: ## Deploy to VPS (triggers GitHub Actions)
@echo "Push to main to trigger deployment, or run:"
@echo " gh workflow run deploy-docker.yml"
init-submodules: ## Initialize all submodules
git submodule update --init --recursive
status: ## Show submodule status
git submodule status
+41 -166
View File
@@ -1,181 +1,56 @@
# Asepharyana Hub
# Asepharyana Infra
Hub repo untuk ekosistem portfolio dan layanan pendukung milik Asep Haryana Saputra.
Aplikasi dipisah sebagai submodule agar frontend, API, dan service pendukung bisa dikembangkan serta di-deploy secara independen.
Reverse-proxy & infrastructure config for [orangevps](https://asepharyana.my.id) (45.127.35.244).
## Services
> **Status (2026-08-28):** Repo ini dulunya monorepo `asepharyana-hub` dengan submodule aplikasi.
> Kini **murni repo infra**: Caddy reverse proxy (source of truth), firewall, drop-in systemd,
> dan docs. Build + deploy tiap aplikasi pindah ke repo masing-masing (self-contained CI).
| Service | Path | Notes |
| :------ | :------------- | :----------------------------- |
| Scraper | `apps/scraper` | Web scraper service + Dapr SDK |
| NATS | — | Message broker + JetStream |
| Dapr | — | Sidecar runtime (per service) |
## Repositori Aplikasi (self-contained build & deploy)
## Infrastructure
| Repo | Deskripsi | Deploy unit |
|------|-----------|-------------|
| [`asepharyana/hub`](https://github.com/asepharyana/hub) | Portfolio SPA (Next.js, port 4003, dashboard) | `hub` |
| [`asepharyana/scraper`](https://github.com/asepharyana/scraper) | Rust/Axum scraper API (port 4091) | `scraper` |
| [`asepharyana/llm-api`](https://github.com/asepharyana/llm-api) | Rust LLM API (llama.cpp, port 8080) | `llm-api` |
File compose berada di `infra/compose/`:
Tiap repo punya `flake.nix` + `.github/workflows/deploy.yml` sendiri:
`nix build .#<pkg>` → **push ke Attic binary cache** (`attic.asepharyana.my.id/asepharyana`) → VPS substitute via `nix-store --realise` → `nix-env --profile` → `systemctl restart`.
Push ke `main` (atau `workflow_dispatch`) langsung deploy; tidak ada lagi pointer submodule.
- `traefik.yml`: reverse proxy Traefik untuk semua layanan.
- `shared.yml`: Redis (cache + Dapr state store).
- `nats.yml`: NATS message broker dengan JetStream persistence.
- `dapr.yml`: Dapr placement service untuk koordinasi sidecar.
- `scraper.yml`: manifest deploy per service (app + Dapr sidecar).
## Infra di Repo Ini
Dockerfile per service berada di `infra/docker/`.
| Path | Isi |
|------|-----|
| `infra/caddy/Caddyfile.prod` | **Source of truth** `/etc/caddy/Caddyfile` (auto-deploy via CI) |
| `infra/firewall/firewall.sh` | deny-by-default iptables (SSH/80/443/4013/Tailscale/TCPShield) |
| `infra/firewall/99-*.conf` | sysctl hardenings |
| `infra/prometheus/targets.yml` | file_sd targets |
| `infra/systemd/scraper-otel.conf` | drop-in OTEL untuk scraper service |
| `docs/` | arsitektur + operasional (VPS) |
## Docker Image Builds
## CI/CD
Build image via Dockerfile:
| Workflow | Trigger | Aksi |
|----------|---------|------|
| `caddy-deploy.yml` | push main menyentuh `infra/**`, atau manual | sync `Caddyfile.prod` → `/etc/caddy/Caddyfile` → reload → verifikasi rute |
## Local Setup / Snapshot VPS
```bash
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest .
# Clone infra repo
git clone https://github.com/asepharyana/infra.git
# Diff config live vs repo
diff /etc/caddy/Caddyfile infra/caddy/Caddyfile.prod
# Koneksi VPS (public)
ssh code@45.127.35.244
```
Tag and push:
## Menambahkan Service Baru / Subdomain
```bash
SHORT_SHA=$(git rev-parse --short HEAD)
1. Aplikasi punya repo sendiri + `deploy.yml` (lihat template di repo app yang ada).
2. Registrasi unit systemd di VPS (manual/ops) → app jalan di port lokal.
3. Tambah site block di `infra/caddy/Caddyfile.prod` (pola `import proxy <port>`) → push → CI reload Caddy.
4. (Opsional) Tambah unit ke `MONITORED_UNITS` dashboard hub di repo `asepharyana/hub`.
docker tag scraper-api:latest ghcr.io/asepharyana/asepharyana-hub/scraper-api:sha-$SHORT_SHA
docker push ghcr.io/asepharyana/asepharyana-hub/scraper-api:sha-$SHORT_SHA
```
## Local Development
### 1) Jalankan dependency bersama
```bash
docker compose -f infra/compose/shared.yml up -d
```
### 2) Jalankan service yang dibutuhkan
Refer to each service's own documentation for development setup.
## API Docs and Monitoring
Refer to each service's own documentation for API docs.
## Deployment Notes
- Pipeline memakai image tag berbasis commit SHA (`sha-<short-sha>`), bukan `latest`.
- Deploy Compose sekarang mencakup `infra/compose/*.yml` dan `deploy-docker.yml` akan berjalan langsung ketika `infra/compose/**` berubah.
## Networking & Tailscale
### Arsitektur
Semua VPS terhubung via **Tailscale**. Setiap VPS punya IP Tailscale dan service berkomunikasi antar VPS melalui Tailscale network (`100.64.0.0/10`).
| VPS | Tailscale IP | Service |
| :--------- | :-------------- | :------------------------------------- |
| `imrnes` | `100.121.180.82` | PostgreSQL, Redis |
| `orangevps` | `100.79.111.61` | App containers (Traefik, scraper-api) |
| `archlinux` | `100.84.39.83` | _(development machine)_ |
### Container → Tailscale Connectivity
Docker containers di bridge network (`app-shared-net`) **tidak otomatis bisa access Tailscale IPs** karena Tailscale menggunakan **custom policy routing** (routes di `table 52`, bukan `main` table).
#### Fix: Tailscale Route di Main Table
Agar container bisa reach Tailscale IPs (untuk DB, Redis, dll), tambahkan route ke `main` routing table:
```bash
# Manual (hilang setelah reboot)
ip route add 100.64.0.0/10 dev tailscale0 table main
# Persistent (systemd service)
# Sudah dikonfigurasi sebagai /etc/systemd/system/tailscale-routes.service
# Service ini berjalan otomatis setelah tailscaled start
systemctl enable tailscale-routes.service
systemctl start tailscale-routes.service
```
#### Environment Variables
Service yang connect ke Tailscale IP:
```env
# PostgreSQL di imrnes
DATABASE_URL=postgres://user:pass@100.121.180.82:5432/dbname
# Redis di imrnes
REDIS_URL=redis://100.121.180.82:6379
```
#### Persistent Systemd Service
File: `/etc/systemd/system/tailscale-routes.service`
```ini
[Unit]
Description=Add Tailscale routes to main routing table
After=tailscaled.service
Requires=tailscaled.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/sbin/ip route add 100.64.0.0/10 dev tailscale0 table main 2>/dev/null || /usr/sbin/ip route replace 100.64.0.0/10 dev tailscale0 table main'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
```
Install & enable:
```bash
sudo tee /etc/systemd/system/tailscale-routes.service > /dev/null << 'EOF'
[Unit]
Description=Add Tailscale routes to main routing table
After=tailscaled.service
Requires=tailscaled.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/sbin/ip route add 100.64.0.0/10 dev tailscale0 table main 2>/dev/null || /usr/sbin/ip route replace 100.64.0.0/10 dev tailscale0 table main'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable tailscale-routes.service
sudo systemctl start tailscale-routes.service
```
#### Troubleshooting
```bash
# Cek Tailscale peers
tailscale status
# Cek route table 52 (Tailscale internal)
ip route show table 52
# Cek route table main (yang dipakai container)
ip route show table main | grep 100.
# Test connectivity dari dalam container
docker exec <container> node -e "
const net = require('net');
const c = new net.Socket();
c.setTimeout(5000);
c.connect(5432, '100.121.180.82', () => { console.log('OK'); c.end(); });
c.on('error', e => { console.log('FAIL:', e.code); });
c.on('timeout', () => { console.log('TIMEOUT'); c.destroy(); });
"
# Cek service tailscale-routes
systemctl status tailscale-routes.service
```
## Menambahkan Aplikasi Baru
Panduan langkah demi langkah untuk menambahkan aplikasi baru ada di `docs/add-new-app.md`.
## License
MIT
Lihat `docs/add-new-app.md` untuk detail.
Submodule apps/hub deleted from 51798f483f
Submodule apps/llm-api deleted from d8425ea3b3
Submodule apps/scraper deleted from bc782ae4f8
Submodule apps/tools deleted from 3956b90c3c
-73
View File
@@ -1,73 +0,0 @@
{
"$schema": "https://biomejs.dev/schemas/2.5.3/schema.json",
"assist": { "actions": { "source": { "organizeImports": "on" } } },
"linter": {
"enabled": true,
"rules": {
"preset": "recommended",
"a11y": {
"useButtonType": "warn",
"useIframeTitle": "warn",
"noSvgWithoutTitle": "warn",
"useAltText": "warn"
},
"complexity": {
"noForEach": "off",
"noStaticOnlyClass": "off",
"noThisInStatic": "off",
"useOptionalChain": "off"
},
"suspicious": {
"noArrayIndexKey": "warn",
"noConsole": "off",
"noExplicitAny": "warn"
},
"correctness": {
"noUnusedVariables": "error",
"useParseIntRadix": "off"
},
"style": {
"noNonNullAssertion": "off"
}
}
},
"formatter": {
"enabled": true,
"formatWithErrors": false,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100,
"lineEnding": "lf"
},
"javascript": {
"jsxRuntime": "transparent",
"formatter": {
"quoteStyle": "single",
"jsxQuoteStyle": "double",
"trailingCommas": "all",
"semicolons": "always",
"arrowParentheses": "always"
}
},
"css": {
"parser": {
"tailwindDirectives": true
}
},
"files": {
"ignoreUnknown": false,
"includes": [
"**",
"!**/.opencode",
"!**/dist",
"!**/out",
"!**/build",
"!**/node_modules",
"!**/target",
"!**/coverage",
"!**/*.env",
"!**/*.env.*",
"!**/apps/react/src/routeTree.gen.ts"
]
}
}
-31
View File
File diff suppressed because one or more lines are too long
-238
View File
@@ -1,238 +0,0 @@
# Arsitektur asepharyana-hub
## Topologi Fisik
Dua node terhubung via **Tailscale** overlay network:
```
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ orangevps (VPS) │ │ imrnes (Bare-metal) │
│ IP: 45.127.35.244 │ │ Tailscale: 100.121.180.82 │
│ Tailscale: 100.x.x.x │◄──────┤ │
│ │ │ Layanan: │
│ Layanan: │ │ ├─ PostgreSQL (port 6432) │
│ ├─ Traefik (port 80/443) │ │ └─ Redis (port 6379) │
│ ├─ NATS + JetStream │ │ │
│ ├─ Dapr Placement │ └──────────────────────────────┘
│ ├─ Redis (cache, Dapr) │
│ ├─ Scraper API + Dapr │
│ └─ Hub (Next.js SPA) │
└──────────────────────────────┘
```
### Konektivitas Container ke Tailscale
Container di `orangevps` tidak bisa langsung mencapai IP Tailscale (`100.x.x.x`). Route Tailscale harus ditambahkan ke tabel routing utama (`main`) via `tailscale-routes.service` agar traffic dari container bisa melewati host ke Tailscale.
## Alur Request HTTP (External)
```
Internet
│
▼ Port 443
Traefik (v3.6)
├─ TLS termination (sertifikat dari volume mount)
├─ Middleware chain: secure-headers → compress → retry → rate-limit → buffer
├─ Plugin: real-ip (Cloudflare), block-sensitive-paths
│
▼ Router matching
Host(`asepharyana.my.id`) || Host(`www.asepharyana.my.id`) → hub
host(`hub.asepharyana.my.id`) → hub (SPA + dashboard)
Host(`scraper.asepharyana.my.id`) || Host(`api.asepharyana.my.id`) → scraper-api
│
├─ hub (Next.js, port 3000)
│ ├─ / — Portfolio SPA
│ ├─ /dashboard — Ops dashboard (client-side, auto-refresh 15s)
│ ├─ /api/dashboard — JSON: Docker containers, Jaeger traces, Prometheus metrics
│ └─ Docker socket mounted (:ro) for container discovery
│
▼ Service load balancer
http://scraper-api:4091
│
▼
Scraper API (Rust / Axum)
├─ Health check: GET /, respon 200
├─ REST endpoints
├─ Database via `DATABASE_URL` (Tailscale → PostgreSQL di imrnes)
├─ Cache via `REDIS_URL` (Redis lokal di container)
└─ Pub/sub via Dapr sidecar (localhost:3500)
```
## Infrastruktur Internal
### Docker Compose Project
Semua service berjalan dalam satu Docker Compose project bernama `compose` dan bergabung di network `app-shared-net`:
| File | Service | Peran |
|------|---------|-------|
| `traefik.yml` | `traefik` | Reverse proxy + TLS + metrics Prometheus |
| `shared.yml` | `redis` | Cache, session store, backend Dapr pub/sub & state |
| `nats.yml` | `nats` | Message broker + JetStream persistent streaming |
| `dapr.yml` | `dapr-placement` | Koordinasi actor placement untuk sidecar Dapr |
| `scraper.yml` | `scraper-api` + `scraper-api-dapr` | Aplikasi Rust + sidecar Dapr |
| `hub.yml` | `hub` | Next.js SPA portfolio + dashboard + Docker socket |
| `observability.yml` | `otel-collector`, `jaeger`, `prometheus`, `node-exporter` | Tracing, metrics, observability |
### Dapr Sidecar Pattern
Setiap aplikasi yang menggunakan Dapr mendapat sidecar container `daprd`:
```
┌─────────────────────┐
│ scraper-api │
│ (app port 4091) │
└────────┬────────────┘
│ localhost:3500 (HTTP)
│ localhost:50001 (gRPC)
┌────────▼────────────┐
│ scraper-api-dapr │
│ (daprd sidecar) │
│ │
│ Dapr components: │
│ ├─ pubsub.redis │
│ └─ state.redis │
└─────────────────────┘
```
Komponen Dapr:
| Komponen | Tipe | Backend |
|----------|------|---------|
| `pubsub` | `pubsub.redis` | `redis:6379` |
| `statestore` | `state.redis` | `redis:6379` (prefix `dapr`) |
### Monitoring & Auto-Discovery
#### Prometheus Docker Auto-Discovery
Prometheus menggunakan `docker_sd_configs` untuk auto-detect container yang perlu di-scrape. Cukup tambah label pada container:
```yaml
labels:
- 'prometheus.io/scrape=true'
- 'prometheus.io/port=8080' # port metrics endpoint
- 'prometheus.io/path=/metrics' # optional, default /metrics
```
Prometheus akan auto-detect dan mulai scrape container dalam 15 detik.
#### Traefik Metrics
Traefik mengekspos metrics Prometheus di port 8080 (`--metrics.prometheus=true`). Metrics yang tersedia:
| Metric | Query untuk dashboard |
|--------|----------------------|
| Request rate | `sum(rate(traefik_service_requests_total[1m]))` |
| Latency | `avg(traefik_service_request_duration_seconds_sum / traefik_service_request_duration_seconds_count) * 1000` |
| Error rate | `sum(rate(traefik_service_requests_total{code=~"5.."}[1m]))` |
Dashboard di `/api/dashboard` returns node metrics + Traefik range data untuk 4 sparkline charts (RPS, latency, errors, trace volume).
#### Docker Socket Access
Container yang perlu akses Docker socket (`/var/run/docker.sock`) harus punya group docker (GID 988):
```yaml
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
group_add:
- '988'
```di compose atau `--group-add 988` via CLI. Berlaku untuk `hub` (container list) dan `prometheus` (Docker SD).
### NATS + JetStream
NATS berjalan dengan flag `-js` untuk mengaktifkan JetStream. Persistent stream disimpan di volume `nats_data`. Dapr pub/sub routing:
```
Service → Dapr sidecar (pubsub.redis) → Redis streams
```
> **Catatan:** Saat ini Dapr pub/sub menggunakan Redis, bukan NATS. Jika ingin migrasi ke NATS untuk pub/sub, komponen Dapr perlu diganti dengan `pubsub.nats`.
## Arsitektur CI/CD
```
Push ke main (apps/**, infra/**)
│
▼
docker-build-push.yml
├─ Phase 1: Detect changed services
├─ Phase 2: Build & Push image ke GHCR
└─ Phase 3: Update compose manifest + submodule pointer
│
▼ (workflow_run trigger)
deploy-docker.yml
├─ SSH ke orangevps
├─ Git sync, pull images
├─ Remove stale containers
└─ Selective restart service
```
Submodule update dari remote repo via `repository_dispatch`:
```
Push ke asepharyana-hub-scraper
│
▼ (repository_dispatch)
update-submodule.yml
├─ Update submodule pointer
└─ Commit & push ke hub repo
│
▼ (repository_dispatch trigger)
docker-build-push.yml
└─ Build, push, deploy
```
## Image Tagging Strategy
| Tag | Contoh | Penggunaan |
|-----|--------|------------|
| `sha-<short>` | `sha-a3c5d74` | Immutable, deterministic rollback |
| `latest` | `latest` | Mutable, convenience |
| `buildcache` | `sha-a3c5d74-buildcache` | Registry-based build cache (internal) |
## Networking
### Port Map
| Port | Service | Deskripsi |
|------|---------|-----------|
| 443 | Traefik | HTTPS eksternal |
| 80 | Traefik | Redirect ke HTTPS |
| 4222 | NATS | Client connections |
| 8222 | NATS | HTTP monitor / health |
| 6379 | Redis | Internal container network |
| 3500 | Dapr sidecar | Dapr HTTP API (per service) |
| 50001 | Dapr sidecar | Dapr gRPC API (per service) |
| 50005 | Dapr placement | Actor placement |
| 4091 | Scraper API | Aplikasi HTTP |
## Event Topics Convention
Semua event menggunakan prefix `hub.`:
| Topic | Payload | Deskripsi |
|-------|---------|-----------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
## Service Registry (Traefik)
Domain routing:
| Subdomain | Service | URL Backend |
|-----------|---------|-------------|
| `asepharyana.my.id` (root) | Hub SPA + dashboard | `http://hub:3000` |
| `www.*` | Hub (alias) | `http://hub:3000` |
| `hub.*` | Hub (alias) | `http://hub:3000` |
| `scraper.*` | Scraper API | `http://scraper-api:4091` |
| `api.*` | Scraper API (alias) | `http://scraper-api:4091` |
| `traefik.*` | Traefik Dashboard | `api@internal` |
| `jaeger.*` | Jaeger UI | `http://jaeger:16686` |
Semua domain tersedia di:
- `<service>.asepharyana.my.id`
- `<service>.asepharyana.web.id`
-681
View File
@@ -1,681 +0,0 @@
# Deployment Guide
Panduan deploy aplikasi apapun menggunakan **Docker + Docker Compose + GitHub Actions + VPS**.
## Arsitektur
```
GitHub Repo ──► GitHub Actions ──► Registry (GHCR / Docker Hub / ECR / dll.)
│
▼
VPS (<VPS_HOST>)
docker compose pull + up
```
## Prerequisites
- Docker Engine >= 24.x
- Docker Compose v2 (plugin)
- Git
- Akun GitHub dengan akses repo
- SSH key di `~/.ssh/<KEY_NAME>` (default: `id_ed25519`)
## Konfigurasi VPS Target
Buat berkas `~/orangevps` (atau sesuaikan dengan env Anda):
```text
ssh <USER>@<VPS_HOST>
```
Contoh isi `~/orangevps`:
```text
ssh root@45.127.35.244
```
| Parameter | Nilai | Contoh |
|-----------|-------|--------|
| User | `<USER>` | `root` |
| Host | `<VPS_HOST>` | `45.127.35.244` |
| SSH Key | `~/.ssh/<KEY_NAME>` | `~/.ssh/id_ed25519` |
| Target Dir di VPS | `<VPS_TARGET_DIR>` | `/opt/app` atau `/root/app` |
> Tip: Jika SSH key menggunakan nama selain default, sesuaikan path dan `ssh -i` sesuai.
## Registry
Pilih registry untuk menyimpan image Docker. Sesuaikan dengan proyek:
| Registry | URL | Auth |
|----------|-----|------|
| GitHub Container Registry | `ghcr.io` | `GITHUB_TOKEN` |
| Docker Hub | `docker.io` | username / PAT |
| AWS ECR | `<account>.dkr.ecr.<region>.amazonaws.com` | `aws ecr get-login-password` |
| Google GCR | `gcr.io` | `gcloud auth print-access-token` |
| Azure ACR | `<registry>.azurecr.io` | `az acr login` |
Contoh namespace untuk GHCR:
```text
Registry : ghcr.io
Namespace: <GITHUB_USERNAME_OR_ORG>
Repo : <REPO_NAME>
```
Pastikan package/visibility di registry mengizinkan akses pull dari VPS.
---
## Deploy Otomatis (Recommended)
Gunakan GitHub Actions untuk otomatisasi build, push, dan deploy.
### Workflow 1: Build dan Push Image
File: `.github/workflows/docker-build-push.yml`
```yaml
name: Build and Push Docker Images
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@v4
- uses: docker/build-push-action@v7
with:
context: .
file: Dockerfile
push: true
tags: |
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:latest
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:sha-${{ github.sha }}
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache,mode=max
```
Ubah `<SERVICE_NAME>` sesuai service (misal: `app`, `web`, `api`). Jika monorepo, gunakan matrix strategy untuk build beberapa service sekaligus.
### Workflow 2: Deploy ke VPS
File: `.github/workflows/deploy-docker.yml`
```yaml
name: Deploy Docker to VPS
on:
workflow_run:
workflows: ['Build and Push Docker Images']
types: [completed]
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Deploy to VPS
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
VPS_TARGET_DIR: ${{ secrets.VPS_TARGET_DIR }}
ENV_FILE_PRODUCTION: ${{ secrets.ENV_FILE_PRODUCTION }}
run: |
set -euo pipefail
mkdir -p ~/.ssh
echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -H -t ed25519,rsa "$VPS_HOST" >> ~/.ssh/known_hosts
SSH_OPTS=(-o ControlMaster=auto -o ControlPath=/tmp/ssh-%r@%h:%p -o ControlPersist=600 -o StrictHostKeyChecking=yes)
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "mkdir -p $VPS_TARGET_DIR && mkdir -p $VPS_TARGET_DIR/infra/compose"
echo "$ENV_FILE_PRODUCTION" > .env.prod
scp "${SSH_OPTS[@]}" .env.prod "$VPS_USER@$VPS_HOST:$VPS_TARGET_DIR/.env"
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" bash -s <<'EOF'
set -euo pipefail
cd "$VPS_TARGET_DIR"
docker network inspect app-shared-net >/dev/null 2>&1 || docker network create app-shared-net
if [ ! -d ".git" ]; then
git init
git remote add origin https://github.com/<GITHUB_USER>/<REPO_NAME>.git
fi
git fetch origin main --depth=1 || true
git reset --hard FETCH_HEAD
docker compose --env-file .env pull
docker compose --env-file .env up -d --remove-orphans
EOF
```
### Secrets GitHub yang Diperlukan
Buka **Settings > Secrets and variables > Actions**:
| Secret | Deskripsi |
|--------|-----------|
| `SSH_PRIVATE_KEY` | Isi dengan `cat ~/.ssh/<KEY_NAME>` |
| `VPS_HOST` | IP atau domain VPS |
| `VPS_USER` | User SSH (misal: `root`, `ubuntu`, `deploy`) |
| `VPS_TARGET_DIR` | Direktori aplikasi di VPS |
| `ENV_FILE_PRODUCTION` | Isi dengan environment production |
### Trigger Manual
```bash
gh workflow run deploy-docker.yml
```
---
## Dockerfile Patterns
Pilih pattern sesuai jenis aplikasi.
### Pattern 1: Multi-stage Build (SPA / static assets)
```dockerfile
FROM oven/bun:1 AS builder
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile
COPY . .
RUN bun run build
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```
### Pattern 2: Single-stage (runtime image)
```dockerfile
FROM oven/bun:1
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile
COPY . .
EXPOSE 3000
CMD ["bun", "run", "start"]
```
### Pattern 3: Compiled binary (Rust / Go / Zig)
```dockerfile
FROM rust:1 AS builder
WORKDIR /app
COPY . .
RUN cargo build --release
FROM debian:bookworm-slim
COPY --from=builder /app/target/release/app /usr/local/bin/app
EXPOSE 8080
CMD ["app"]
```
---
## Docker Compose Patterns
### Single service
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
ports:
- "3000:3000"
environment:
- NODE_ENV=production
```
### Multi-service dengan shared network
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
networks: [app-shared-net]
redis:
container_name: redis
image: redis:7-alpine
restart: always
networks: [app-shared-net]
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dengan reverse proxy (Traefik / Caddy / Nginx)
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
networks: [app-shared-net]
labels:
- 'traefik.enable=true'
- 'traefik.http.routers.app.rule=Host(`app.example.com`)'
- 'traefik.http.routers.app.entrypoints=websecure'
- 'traefik.http.routers.app.tls=true'
- 'traefik.http.services.app.loadbalancer.server.port=3000'
networks:
app-shared-net:
name: app-shared-net
external: true
```
---
## Deploy Manual (Lokal)
### 1. Build dan Push ke Registry
Login ke registry:
```bash
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Build dan push:
```bash
docker build -t ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest -f Dockerfile .
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest
```
Tag tambahan dengan SHA commit:
```bash
SHORT_SHA=$(git rev-parse --short HEAD)
docker tag ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest \
ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
```
### 2. Pull dan Deploy di VPS
SSH ke VPS:
```bash
ssh -i ~/.ssh/<KEY_NAME> <USER>@<VPS_HOST>
```
Clone repo (jika belum):
```bash
git clone https://github.com/<GITHUB_USER>/<REPO_NAME>.git <VPS_TARGET_DIR>
cd <VPS_TARGET_DIR>
```
Buat shared network (hanya sekali):
```bash
docker network create app-shared-net
```
Siapkan environment:
```bash
cp .env.example .env
# Edit .env sesuai nilai production
nano .env
```
Login ke registry di VPS:
```bash
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Pull gambar terbaru:
```bash
cd <VPS_TARGET_DIR>
docker compose -f docker-compose.yml --env-file .env pull
```
Deploy (up):
```bash
docker compose -f docker-compose.yml --env-file .env up -d --remove-orphans
```
Verifikasi:
```bash
docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs -f <SERVICE_NAME>
```
---
## Deployment Order (Manual)
Jika deploy bertahap, gunakan urutan ini:
```bash
# 1. Shared services (Redis, database, dll.)
docker compose -f infra/compose/shared.yml up -d
# 2. Reverse proxy
docker compose -f infra/compose/traefik.yml up -d
# 3. Aplikasi
docker compose \
-f infra/compose/app1.yml \
-f infra/compose/app2.yml \
up -d
```
---
## Perintah Berguna di VPS
```bash
# Lihat semua container
docker ps -a
# Log service
docker logs -f <container_name>
# Restart satu service
docker compose -f <compose_file> up -d --force-recreate
# Hapus network lama (hati-hati)
docker network rm app-shared-net
docker network create app-shared-net
# Bersihkan image unused
docker image prune -a -f
docker system prune -a -f
```
---
## Troubleshooting
### Image tidak bisa di-pull
Pastikan sudah login ke registry di VPS:
```bash
docker logout ghcr.io
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Periksa visibility package di registry (harus `Public` atau akses diberikan).
### Port sudah dipakai
```bash
docker ps | grep :80
docker ps | grep :443
```
### Reverse proxy tidak routing
Periksa label di compose file dan pastikan shared network ada:
```bash
docker network inspect app-shared-net
docker logs traefik
```
---
## Environment Variable Management
### Pola 1: `.env` di VPS (recommended untuk production)
```bash
# Di VPS
cd <VPS_TARGET_DIR>
cp .env.example .env
# Edit sesuai production
nano .env
```
CI/CD upload `.env` via secret, tidak simpan di repo.
### Pola 2: Docker secrets (Swarm mode)
```yaml
services:
app:
image: app:latest
secrets:
- db_password
secrets:
db_password:
file: ./secrets/db_password.txt
```
### Pola 3: External secret manager
- **HashiCorp Vault**: inject via env atau file
- **AWS Secrets Manager**: `aws secretsmanager get-secret-value`
- **Doppler / Infisical**: unified secret management
---
## Tagging dan Versioning
### Strategy yang umum
| Strategy | Contoh tag | Kegunaan |
|----------|-----------|----------|
| Latest + SHA | `latest`, `sha-abc1234` | CI/CD cepat, traceable |
| SemVer | `1.2.3`, `1.2`, `1` | Release publik |
| Git tag mirror | `v1.2.3` | Sync dengan git tag |
| Branch mirror | `main`, `develop` | Preview / staging |
### Contoh git tag driven deploy
```bash
git tag v1.2.3
git push origin v1.2.3
```
CI/CD membaca tag, build image dengan tag yang sama, dan deploy.
---
## Rollback
### Rollback via registry
```bash
# Lihat tag yang tersedia
docker manifest inspect ghcr.io/org/app:latest
# atau lihat UI registry
# Di VPS, edit compose file ke tag sebelumnya
# lalu:
docker compose --env-file .env pull
docker compose --env-file .env up -d --remove-orphans
```
### Rollback via git
```bash
git revert HEAD
git push origin main
# CI/CD otomatis build dan deploy versi sebelumnya
```
---
## Health Checks
### Di Dockerfile
```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost:3000/health || exit 1
```
### Di Docker Compose
```yaml
services:
app:
image: app:latest
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
interval: 30s
timeout: 3s
retries: 3
start_period: 10s
```
---
## Monitoring & Observability
```bash
# Log aggregated
docker compose logs -f --tail=100
# Resource usage
docker stats
# Disk usage
docker system df
# Cleanup
docker system prune -a -f
```
---
## Catatan Keamanan
- Jangan commit `.env` atau SSH private key ke repo.
- Gunakan GitHub Secrets (atau secret manager) untuk credential di CI/CD.
- Rotate token dan key secara berkala.
- Batasi akses SSH ke VPS (ubah port default, gunakan fail2ban).
- Set `StrictHostKeyChecking=yes` pada SSH opsional deployment.
---
---
## Proyek Ini: asepharyana-hub
> Dokumentasi spesifik untuk repo ini. Lihat juga [ADR-0002](adr/0002-env-file-via-github-secret.md).
### Topologi
| Host | IP | Peran |
|------|----|-------|
| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr |
| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379) |
### Environment Variables
**Production `.env` tidak pernah di-commit.** File ini disimpan sebagai GitHub secret `ENV_FILE_PRODUCTION` dan di-SCP ke VPS saat deploy via `deploy-docker.yml`.
Cara update:
```bash
# Baca current .env dari VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
# Update GitHub secret (dari output di atas)
cat > /tmp/env-updated << 'EOF'
<paste content, edit, lalu>
EOF
cat /tmp/env-updated | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
```
**Jangan manual edit `.env` di VPS tanpa update GitHub secret juga** — nanti ke- overwrite pas deploy berikutnya.
### Database
| Variable | Value |
|----------|-------|
| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` |
| `REDIS_URL` | `redis://redis:6379` (Docker network) |
### Kompose
Proyek compose bernama `compose`, terdiri dari 5 file yang selalu di-include bersamaan:
```bash
/root/asepharyana-hub/infra/compose/
├── traefik.yml # Reverse proxy
├── shared.yml # Redis
├── nats.yml # NATS
├── dapr.yml # Dapr placement
├── scraper.yml # Scraper API
└── observability.yml # OTel Collector, Jaeger, Dashboard
```bash
cd /root/asepharyana-hub
docker compose \
-p compose \
--env-file .env \
-f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/scraper.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/observability.yml \
up -d --remove-orphans
```
---
## Checklist Deploy Proyek Baru
1. [ ] Dockerfile ditest lokal (`docker build`, `docker run`)
2. [ ] Docker Compose file valid (`docker compose config`)
3. [ ] `.dockerignore` sesuai (node_modules, .git, .env)
4. [ ] Registry dibuat (GHCR package / Docker Hub repo / ECR / dll.)
5. [ ] GitHub Actions workflow dibuat dengan permission `packages: write`
6. [ ] VPS siap: Docker, Docker Compose, SSH key
7. [ ] Shared network dibuat (`docker network create`)
8. [ ] `.env` production di-VPS atau via secret manager
9. [ ] Reverse proxy (Traefik / Caddy / Nginx) routing ke container
10. [ ] Health check endpoint aktif
-206
View File
@@ -1,206 +0,0 @@
# Development Guide
Panduan setup lingkungan development lokal untuk kontributor `asepharyana-hub`.
## Prasyarat
| Tool | Versi Minimal | Catatan |
|------|---------------|---------|
| Git | 2.40+ | Submodule support |
| Docker | 24+ | Dengan Docker Compose v2 plugin |
| Rust | 1.85+ | Hanya untuk `apps/scraper` |
| Bun | 1.x | Root tooling (Biome) |
| Dapr CLI | 1.14+ | Opsional, untuk development dengan Dapr |
## Setup Awal
```bash
# 1. Clone repo
git clone https://github.com/asepharyana/asepharyana-hub.git
cd asepharyana-hub
# 2. Init submodules
make init-submodules
# 3. Setup environment
cp .env.example .env
# Edit .env sesuai kebutuhan lokal
# 4. Install root dependencies
bun install
```
## Menjalankan Infrastruktur Lokal
Beberapa service membutuhkan Redis. Jalankan dengan:
```bash
make dev
# atau equivalen:
docker compose -f infra/compose/shared.yml up -d
```
Ini akan menjalankan Redis Alpine di `localhost:6379`.
### (Opsional) NATS Lokal
Jika service membutuhkan pub/sub:
```bash
docker compose -f infra/compose/nats.yml up -d
# NATS client: localhost:4222
# NATS monitor: localhost:8222
```
### (Opsional) Dapr Placement Lokal
Jika service membutuhkan sidecar Dapr:
```bash
docker compose -f infra/compose/dapr.yml up -d
# Dapr placement: localhost:50005
```
## Menjalankan Service Lokal
### Scraper API (Rust)
```bash
# Pastikan Redis sudah running (make dev)
cd apps/scraper
# Cargo run
cargo run
# Dengan Dapr sidecar (jika placement running)
dapr run \
--app-id scraper-api \
--app-port 4091 \
--dapr-http-port 3500 \
--resources-path ../../infra/dapr/components \
-- cargo run
```
### Dengan Docker Compose (Full Stack)
Untuk menjalankan semua service sekaligus:
```bash
docker compose \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env \
up -d
```
Untuk service baru, tambahkan compose file-nya ke daftar.
## Update Submodules
### Pull latest dari semua submodule
```bash
make update-submodules
# atau:
git submodule update --remote --merge --recursive
```
### Check status submodule
```bash
make status
# atau:
git submodule status
```
### Sync .env ke submodule
```bash
bash scripts/2updateenv.sh
# Copy .env root ke apps/*/
```
## Linting & Formatting
Root repo menggunakan **Biome** untuk linting dan formatting:
```bash
bun run check # Lint + format + write
bun run ci # CI mode (no write, exit code on issues)
bun run lint # Lint only
bun run format # Format only
```
## Build Docker Image Lokal
```bash
# Scraper API
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:local .
# Service baru: tambahkan Dockerfile di infra/docker/
```
## Testing
Saat ini belum ada test runner di root level. Masing-masing submodule mengelola testing sendiri:
```bash
# Scraper API (Rust)
cd apps/scraper && cargo test
```
## Validasi YAML
Sebelum commit perubahan infra, validasi semua file YAML:
```bash
python -c "
import pathlib, yaml
for p in pathlib.Path('infra').rglob('*.yml'):
with open(p) as f: yaml.safe_load(f)
print(f'OK {p}')
for p in pathlib.Path('infra').rglob('*.yaml'):
with open(p) as f: yaml.safe_load(f)
print(f'OK {p}')
"
for f in infra/compose/*.yml; do
docker compose -f "$f" config >/dev/null && echo "OK $f"
done
```
## Git Workflow
### Commit Convention
```
<type>(<scope>): <description>
```
Type: `feat`, `fix`, `chore`, `docs`, `refactor`, `test`, `ci`, `perf`, `style`
Scope: `scraper`, `infra`, `ci`, `dapr`, `nats`, `docs`, `deps`, `scripts`, `root`
Contoh:
```
feat(scraper): add image cache endpoint
fix(infra): correct Traefik rate-limit config
chore(deps): bump biome to 2.5.0
```
### Branch Strategy
- `main` — production branch, push triggers CI/CD
- Fitur baru: branch dari `main`, PR ke `main`
- Submodule development: dilakukan di repo masing-masing, hub hanya update pointer
## Deployment ke VPS
Push ke `main` otomatis trigger CI/CD. Untuk trigger manual:
```bash
gh workflow run deploy-docker.yml
```
Lihat `docs/DEPLOYMENT.md` untuk detail.
-162
View File
@@ -1,162 +0,0 @@
# Menambahkan Dapr ke Service Baru
Panduan integrasi Dapr runtime sidecar untuk service di `asepharyana-hub`.
## Prasyarat
- NATS server berjalan (`infra/compose/nats.yml`)
- Dapr placement service berjalan (`infra/compose/dapr.yml`)
## 1. Compose File
Setiap service butuh sidecar container Dapr. Contoh:
```yaml
services:
app:
container_name: app
image: ghcr.io/asepharyana/asepharyana-hub/app:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
networks:
app-shared-net:
aliases:
- app
env_file:
- ../../.env
app-dapr:
container_name: app-dapr
image: daprio/daprd:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
networks:
- app-shared-net
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=app'
- '--app-port=3000'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--config=/dapr/config.yaml'
- '--resources-path=/dapr/components'
volumes:
- ../../infra/dapr:/dapr:ro
networks:
app-shared-net:
name: app-shared-net
external: true
```
## 2. Mengakses Dapr dari Service
### Via HTTP API (semua bahasa)
Sidecar listen di `localhost:3500`:
```bash
# Publish event
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.event.type \
-H "Content-Type: application/json" \
-d '{"key": "value"}'
# Service invocation
curl http://localhost:3500/v1.0/invoke/app/method/endpoint
# State store
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
```
### Via Dapr SDK (Rust)
Tambah ke `Cargo.toml`:
```toml
dapr-sdk = { version = "0.15", features = ["pubsub", "http"] }
tokio-stream = "0.1"
```
Contoh publish event:
```rust
use dapr_sdk::client::{Client, Event};
use dapr_sdk::DaprClient;
let client = DaprClient::new("127.0.0.1", 3500).await?;
client.publish_event("pubsub", "hub.image.cached", serde_json::json!({
"original_url": url,
"cdn_url": cdn_url,
})).await?;
```
Contoh subscribe event:
```rust
let mut stream = client.subscribe_events("pubsub", "hub.image.cached").await?;
while let Some(event) = stream.next().await {
let data: MyEvent = serde_json::from_slice(&event.data)?;
// handle event
}
```
## 3. Event Topics Convention
Gunakan prefix `hub.` untuk semua event:
| Topic | Payload | Description |
|-------|---------|-------------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
## 4. Local Development
Untuk development tanpa Docker:
```bash
# 1. Install Dapr CLI
# 2. Init Dapr local
dapr init
# 3. Run service dengan sidecar
dapr run --app-id app --app-port 3000 --dapr-http-port 3500 \
--resources-path ./infra/dapr/components \
-- cargo run
```
## 5. Verifikasi
```bash
# Sidecar health
curl http://localhost:3500/v1.0/healthz
# Publish test event
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.test \
-H "Content-Type: application/json" \
-d '{"test": true}'
# NATS stream stats
curl http://localhost:8222/jszetstream
```
+33 -158
View File
@@ -1,167 +1,42 @@
# Menambahkan Aplikasi Baru ke Deployment
# Menambahkan Service Baru / Subdomain
Dokumen ini menjelaskan langkah menambahkan service baru ke `asepharyana-hub`. Root repo berfungsi sebagai hub: source aplikasi berada di `apps/<nama-app>` sebagai submodule, sedangkan Docker Compose, Traefik, dan workflow deploy tetap berada di root repo.
Panduan untuk menambahkan service baru di ekosistem `asepharyana/infra` (2026-08-28+, pasca monorepo).
## 1. Buat repo aplikasi
## Prinsip
Buat repo baru di GitHub dengan pola nama:
- **Aplikasi hidup di repo sendiri** (`hub`, `scraper`, `llm-api`) dengan
`flake.nix` + `.github/workflows/deploy.yml` mandiri. Repo infra TIDAK berisi kode app.
- Repo infra (`asepharyana/infra`) hanya mengatur **reverse proxy & config VPS**.
```text
https://github.com/asepharyana/asepharyana-hub-<nama-app>.git
## Langkah
1. **Buat repo aplikasi** (contoh pola: `asepharyana/scraper`).
2. **Tambahkan `flake.nix`** di repo app — derivasi Nix (lihat template di repo app yang ada:
Next.js/bun atau Rust/cargo). Nama paket = nama unit systemd.
3. **Tambahkan `.github/workflows/deploy.yml`** (pola `nix build .#<pkg>` → `nix copy ssh://`
→ `nix-env --profile /nix/var/nix/profiles/<pkg> --set` → `systemctl restart <pkg>`).
Secrets yang dibutuhkan: `SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`.
4. **Di VPS**: buat user systemd + unit (mis. `/etc/systemd/system/<app>.service`,
`ExecStart=/usr/local/bin/bws-exec <app> /nix/var/nix/profiles/<app>/bin/<app>`),
pastikan app jalan di port lokal.
5. **Tambah site block** di `infra/caddy/Caddyfile.prod` (pola `import proxy <port>`),
push ke `main` → CI `caddy-deploy.yml` sync + reload + verifikasi rute.
6. **(Opsional)** Tambah unit ke `MONITORED_UNITS` di hub dashboard
(repo `asepharyana/hub`, `src/app/api/dashboard/route.ts`).
## Contoh site block Caddy
```caddyfile
nama-app.asepharyana.my.id {
import proxy <PORT>
}
```
Lalu tambahkan ke root hub sebagai submodule:
## Verifikasi
```bash
git submodule add https://github.com/asepharyana/asepharyana-hub-<nama-app>.git apps/<nama-app>
git submodule update --init --recursive
```
# Dari local
curl -s -o /dev/null -w '%{http_code}\n' https://nama-app.asepharyana.my.id/
## 2. Tambahkan Dockerfile
Tambahkan Dockerfile runtime di `infra/docker/<nama-app>.Dockerfile`.
Gunakan root repo sebagai build context agar Dockerfile bisa mengakses submodule path:
```bash
docker build -f infra/docker/<nama-app>.Dockerfile -t <nama-app>:local .
```
## 3. Tambahkan Compose file
Buat `infra/compose/<nama-app>.yml`:
```yaml
services:
<nama-app>:
container_name: <nama-app>
image: ghcr.io/asepharyana/asepharyana-hub/<nama-app>:sha-<short-sha>
restart: always
networks:
app-shared-net:
aliases:
- <nama-app>
env_file:
- ../../.env
networks:
app-shared-net:
name: app-shared-net
external: true
```
Gunakan `app-shared-net` agar service dapat diakses oleh Traefik dan service lain.
## 3.5. Tambahkan Dapr sidecar (wajib untuk pub/sub)
Setiap service yang ingin menggunakan Dapr pub/sub atau service invocation harus punya sidecar.
Tambah di `infra/compose/<nama-app>.yml`:
```yaml
<nama-app>-dapr:
container_name: <nama-app>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<nama-app>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--config=/dapr/config.yaml'
- '--resources-path=/dapr/components'
volumes:
- ../../infra/dapr:/dapr:ro
```
Pastikan juga app container punya `depends_on` ke dapr-placement, nats, dan otel-collector:
```yaml
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
```
## 4. Tambahkan route Traefik
Update `infra/traefik/dynamic/apps.yaml`:
```yaml
http:
routers:
<nama-app>:
rule: 'Host(`<subdomain>.asepharyana.my.id`) || Host(`<subdomain>.asepharyana.web.id`)'
entryPoints:
- websecure
tls: {}
middlewares:
- common-chain@file
service: <nama-app>-service
services:
<nama-app>-service:
loadBalancer:
servers:
- url: 'http://<nama-app>:<port>'
```
## 5. Update workflow build
Update `.github/workflows/docker-build-push.yml`:
1. Tambahkan path detection untuk `apps/<nama-app>` dan `infra/docker/<nama-app>.Dockerfile`.
2. Tambahkan service ke matrix build.
3. Tambahkan mapping Dockerfile di step `Docker metadata`.
4. Tambahkan mapping compose file dan submodule path di step `Update tags and submodules`.
## 6. Update workflow deploy
Tambahkan compose file baru ke `ALL_COMPOSE_FILES` di `.github/workflows/deploy-docker.yml`:
```bash
infra/compose/<nama-app>.yml
```
## 7. Update dokumentasi
Update file berikut bila service baru mengubah arsitektur publik:
- `README.md`
- `ARCHITECTURE.md`
- `infra/README.md`
- `.gitmodules`
## 8. Validasi
Jalankan validasi YAML dan compose rendering:
```bash
python - <<'PY'
import pathlib, yaml
for path in pathlib.Path('infra').rglob('*.yml'):
with path.open() as fh:
yaml.safe_load(fh)
print(f'OK {path}')
for path in pathlib.Path('infra').rglob('*.yaml'):
with path.open() as fh:
yaml.safe_load(fh)
print(f'OK {path}')
PY
for f in infra/compose/*.yml; do
docker compose -f "$f" config >/dev/null && echo "OK $f"
done
```
# Dari VPS
systemctl status <app>
+22 -33
View File
@@ -1,49 +1,38 @@
# ADR 0001: Use a Hub Repository with App Submodules
# ADR 0001: Infra Repo — Reverse Proxy Config Only (Submodules Removed)
## Status
Accepted
**Superseded** (2026-08-28) — lihat ADR ini sebagai arsip keputusan awal.
## Context
## Context (aslinya)
The project contains multiple independent application services that share one deployment surface: Docker Compose, Traefik routing, GitHub Actions workflows, and operational documentation.
Proyek awal memakai `asepharyana-hub` sebagai monorepo: aplikasi di `apps/<service>` sebagai
git submodule, infra (compose/traefik/dokumen/CI) terpusat di root. Keputusan itu masuk akal
saat semua service berbagi satu deployment surface.
The services should be developed and versioned independently, while deployment infrastructure should remain centralized so production routing and compose manifests stay consistent.
## Decision (aslinya)
## Decision
Gunakan `asepharyana-hub` sebagai root hub: app code submodule, infra + CI di root.
Use `asepharyana-hub` as the root hub repository.
## Superseded By
- Application code lives under `apps/<service>` as Git submodules.
- Infrastructure lives in the root repo under `infra/`.
- Documentation lives in the root repo under `docs/`.
- CI/CD workflows live in the root repo under `.github/workflows/`.
- Root tooling stays minimal: `package.json`, Prettier, ESLint, Makefile helpers, and deployment scripts.
Mulai **2026-08-28** repo dirombak:
Current app submodules:
| Service | Path | Remote |
| ----------- | -------------- | --------------------------------------- |
| Scraper API | `apps/scraper` | `asepharyana/asepharyana-hub-scraper` |
- **Parent `asepharyana-hub` → `asepharyana/infra`** — murni config reverse proxy (Caddy),
firewall, systemd drop-ins, docs. CI hanya untuk deploy Caddy.
- **App repos di-rename & self-contained**: `hub`, `scraper`, `tools`, `llm-api`.
Masing-masing punya `flake.nix` + `.github/workflows/deploy.yml` sendiri
(`nix build → nix copy → nix-env --profile → systemctl restart`).
- **Submodule dihapus** — tidak ada lagi pointer submodule / repository_dispatch chain.
- `update-submodule.yml`, `notify-parent.yml`, matrix `nix-build.yml` dihapus.
## Consequences
### Positive
- Each app can evolve in its own repository.
- The hub pins exact submodule revisions for reproducible deployments.
- Deployment infrastructure remains centralized and easier to audit.
- Root tooling stays lightweight and does not impose one build system on every service.
- CI tiap app independen: push ke repo app langsung build+deploy, tak perlu 2 hop.
- Parent kecil & fokus: diff Caddyfile mudah di-audit.
- Tanpa submodule = tanpa `dubious ownership` / pointer drift / fetchGit pin.
### Negative
- Developers must understand Git submodule workflows.
- Updating a service requires updating the submodule pointer in the hub repo.
- Cross-service changes require coordinating commits across multiple repositories.
### Mitigations
- Keep `.gitmodules` accurate and minimal.
- Use `scripts/sync-submodules.sh` for local checkout consistency.
- Document service-addition steps in `docs/add-new-app.md`.
- Keep GitHub Actions responsible for Docker image builds, compose tag updates, and deployments.
- Koordinasi cross-repo manual (app + Caddy bila perlu port baru).
- Repo lama `asepharyana-hub-*` redirect ke nama baru (GitHub auto).
+9 -5
View File
@@ -1,8 +1,12 @@
# ADR 0002: Production `.env` via GitHub Encrypted Secret
> **LEGACY (2026-08-28):** Repo `asepharyana-hub` sudah dirombak → `asepharyana/infra`.
> Workflow lama yang SCP `.env` ke VPS tidak dipakai lagi (deploy app pindah ke repo masing-masing,
> secrets via Bitwarden `bws-exec`). ADR ini dipertahankan sebagai arsip.
## Status
Accepted
Accepted (archived)
## Context
@@ -33,7 +37,7 @@ Container reads $DATABASE_URL, $JWT_SECRET, etc.
```bash
# 1. Read current content from the VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
ssh root@45.127.35.244 "cat <VPS app dir, e.g. /home/code/hub>/.env"
# 2. Pipe updated content to the GitHub secret
# (requires gh CLI with repo access)
@@ -43,7 +47,7 @@ cat /path/to/updated-env | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/
gh workflow run deploy-docker.yml
# OR apply immediately on the VPS (for hotfix):
ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.env"
ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' <VPS app dir, e.g. /home/code/hub>/.env"
# Then restart affected containers
```
@@ -69,7 +73,7 @@ ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.e
The VPS runs a single Docker Compose project named `compose` composed of multiple files:
```bash
/root/asepharyana-hub/infra/compose/
<VPS app dir, e.g. /home/code/hub>/infra/compose/
├── traefik.yml # Reverse proxy (TLS termination, routing)
├── shared.yml # Redis
├── nats.yml # NATS message broker + JetStream
@@ -99,7 +103,7 @@ docker compose \
| `SSH_PRIVATE_KEY` | SSH key for VPS access |
| `VPS_HOST` | `45.127.35.244` |
| `VPS_USER` | `root` |
| `VPS_TARGET_DIR` | `/root/asepharyana-hub` |
| `VPS_TARGET_DIR` | `<VPS app dir, e.g. /home/code/hub>` |
| `ENV_FILE_PRODUCTION` | Full `.env` content for production |
## Consequences
@@ -0,0 +1,47 @@
# ADR 0003: Rename Repositori & Pisahkan CI per Aplikasi
## Status
Accepted (2026-08-28)
## Context
Monorepo `asepharyana-hub` (app submodule + infra + CI terpusat) punya kelemahan:
- CI build+deploy semua app menyatu di parent (`nix-build.yml` matrix) — setiap push app
butuh 2 hop (notify-parent → update-submodule → nix-build), rawan drift pointer.
- Nama `asepharyana-hub` ambigu (parent & app prefix sama), dan submodule menambah kompleksitas.
## Decision
Rombak total:
| Lama | Baru | Peran |
|------|------|-------|
| `asepharyana-hub` | `asepharyana/infra` | Reverse proxy (Caddy) + firewall + systemd + docs. CI: caddy-deploy saja. |
| `asepharyana-hub-hub` | `asepharyana/hub` | Portfolio SPA. CI mandiri (deploy.yml). |
| `asepharyana-hub-scraper` | `asepharyana/scraper` | Rust scraper API. CI mandiri. |
| `asepharyana-hub-tools` | `asepharyana/tools` | Tools stack (gateway/workers/frontend). CI mandiri. |
| `asepharyana-hub-llm-api` | `asepharyana/llm-api` | Rust LLM API. CI mandiri. |
| `asepharyana-hub-guide` | `asepharyana/hub-guide` | (tidak di-root; plugin guide — diarsipkan) |
Setiap app repo mendapat:
- `flake.nix` (derivasi build sendiri, tanpa fetchGit submodule)
- `.github/workflows/deploy.yml` (nix build → nix copy → nix-env --profile → systemctl restart)
- Secret `SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`
Parent `infra` mendapat:
- Hapus semua submodule + `update-submodule.yml` + matrix `nix-build.yml`
- `.github/workflows/caddy-deploy.yml` (sync Caddyfile → reload → verify)
- Docs diarahkan ulang.
## Consequences
- **Positif**: CI per-app independen & cepat; parent kecil; tanpa submodule = tanpa fetchGit pin
/ dubious-ownership / pointer churn. Rename GitHub auto-redirect URL lama.
- **Negatif**: koordinasi manual bila app butuh port baru di Caddy; workflow lama di
downstream (skill/cron) perlu update referensi.
## Referensi
- `docs/add-new-app.md` — proses menambah service baru
- `infra/caddy/Caddyfile.prod` — pola site block
-234
View File
@@ -1,234 +0,0 @@
# Backup & Disaster Recovery
## Aset yang Perlu di-Backup
| Aset | Lokasi | Frekuensi | Metode |
|------|--------|-----------|--------|
| Database PostgreSQL | `imrnes` (100.121.180.82:6432) | Harian | `pg_dump` |
| Volume Redis | `orangevps` (Docker volume) | Opsional | Redis RDB / AOF |
| Volume NATS JetStream | `orangevps` (Docker volume) | Opsional | File copy |
| Docker Compose manifests | GitHub (hub repo) | Real-time | Git |
| Environment variables | GitHub secret `ENV_FILE_PRODUCTION` | Manual | `gh secret set` |
| TLS certificates | `orangevps` (`/root/*.pem`, `*.key`) | Saat renew | SCP |
| Tailscale auth | Tailscale admin console | - | Cloud-managed |
| GitHub Actions secrets | GitHub UI | Manual | Backup list |
## Database PostgreSQL (Prioritas Tertinggi)
### Backup Manual
```bash
# Dari orangevps (via Tailscale)
pg_dump -h 100.121.180.82 -p 6432 -U asephs -d hub \
--no-owner --no-acl \
-F c -f /root/db-backups/hub-$(date +%Y%m%d-%H%M%S).dump
# Atau dari imrnes langsung
pg_dump -U asephs -d hub \
-F c -f /backup/hub/hub-$(date +%Y%m%d-%H%M%S).dump
```
### Restore
```bash
# Drop dan recreate database
dropdb -h 100.121.180.82 -p 6432 -U asephs hub
createdb -h 100.121.180.82 -p 6432 -U asephs hub
# Restore dari dump
pg_restore -h 100.121.180.82 -p 6432 -U asephs -d hub \
--no-owner --no-acl \
/path/to/backup/hub-20260101-120000.dump
```
### Backup Otomatis (via Cron di imrnes)
```bash
# /etc/cron.d/hub-db-backup
0 2 * * * root pg_dump -U asephs -d hub -F c -f /backup/hub/hub-$(date +\%Y\%m\%d).dump && find /backup/hub -name "hub-*.dump" -mtime +30 -delete
```
## Volume Docker
### Redis
Redis data bisa di-recover dari NATS events (event sourcing). Jika tidak ada persistence requirement, cukup restart:
```bash
docker volume rm redis_data
docker compose -f infra/compose/shared.yml up -d
```
Jika perlu backup:
```bash
# Save RDB snapshot
docker exec redis redis-cli SAVE
# Copy dari volume
docker run --rm -v redis_data:/data -v /backup:/backup alpine cp /data/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
```
### NATS JetStream
```bash
# Backup volume
docker run --rm -v nats_data:/data -v /backup:/backup alpine \
tar czf /backup/nats-$(date +%Y%m%d).tar.gz -C /data .
```
## Environment Variables
### Backup `.env` dari VPS
```bash
# Simpan current .env dari VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" > .env.backup.$(date +%Y%m%d)
# Update GitHub secret
cat .env.backup.$(date +%Y%m%d) | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
```
### Restore `.env` jika hilang
```bash
# Buat .env baru dari template
cp .env.example .env
# Edit secrets (manual dari password manager atau GitHub secret)
# Atau download dari GitHub secret
gh secret list --repo asepharyana/asepharyana-hub
```
## TLS Certificates
### Backup
```bash
# Di orangevps
tar czf /root/cert-backup-$(date +%Y%m%d).tar.gz \
/root/asepharyana.my.id.pem \
/root/asepharyana.my.id.key \
/root/asepharyana.web.id.pem \
/root/asepharyana.web.id.key \
/root/asepharyana-hub/infra/traefik/dynamic/ssl.yaml
# SCP ke local
scp root@45.127.35.244:/root/cert-backup-*.tar.gz .
```
### Restore
```bash
# SCP ke VPS
scp cert-backup-20260101.tar.gz root@45.127.35.244:/root/
# Extract
ssh root@45.127.35.244 "tar xzf /root/cert-backup-20260101.tar.gz -C / && docker restart traefik"
```
## Disaster Recovery Scenarios
### Skenario 1: VPS (orangevps) mati total
**Dampak:** Semua service down.
**Recovery:**
```bash
# 1. Provision VPS baru (atau restore dari snapshot)
# 2. Install Docker + Tailscale
# 3. Clone repo
git clone https://github.com/asepharyana/asepharyana-hub.git /root/asepharyana-hub
# 4. Setup Tailscale, route service
# 5. Restore .env
echo "<ENV_FILE_PRODUCTION>" > /root/asepharyana-hub/.env
# 6. Restore TLS certs
# 7. Create network
docker network create app-shared-net
# 8. Start services sesuai urutan
cd /root/asepharyana-hub
for f in shared.yml nats.yml dapr.yml traefik.yml scraper.yml; do
docker compose -f infra/compose/$f --env-file .env up -d
done
# 9. Update DNS jika IP baru
```
### Skenario 2: Database (imrnes) mati total
**Dampak:** Semua service yang butuh database error.
**Recovery:**
```bash
# 1. Fix imrnes atau provision server baru
# 2. Setup PostgreSQL
# 3. Restore dari backup terakhir
# 4. Update Tailscale IP jika perlu
# 5. Update .env dan GitHub secret
# 6. Redeploy
```
### Skenario 3: GitHub repository hilang
**Dampak:** Kehilangan CI/CD, tapi Docker images masih ada di GHCR.
**Recovery:**
```bash
# 1. Create repo baru di GitHub
# 2. Push dari local clone
git remote add origin-new https://github.com/asepharyana/asepharyana-hub-new.git
git push origin-new main
# 3. Re-create GitHub secrets
# 4. Re-create workflows
# 5. Update VPS remote
ssh root@45.127.35.244 "cd /root/asepharyana-hub && git remote set-url origin https://github.com/asepharyana/asepharyana-hub-new.git"
```
### Skenario 4: GHCR registry tidak bisa diakses
**Dampak:** Tidak bisa pull image.
**Recovery:**
```bash
# 1. Build image langsung di VPS
docker build -f infra/docker/scraper.Dockerfile -t ghcr.io/asepharyana/asepharyana-hub/scraper-api:local .
# 2. Update compose file untuk sementara
sed -i 's|image: ghcr.io/.*|image: ghcr.io/asepharyana/asepharyana-hub/scraper-api:local|' infra/compose/scraper.yml
# 3. Start
docker compose -f infra/compose/scraper.yml up -d
```
### Skenario 5: Semua server mati (total loss)
**Recovery:**
```bash
# 1. Provision VPS baru
# 2. Provision server database baru
# 3. Setup Tailscale
# 4. Clone repo, restore .env, certs
# 5. Restore database dari backup (jika ada)
# 6. Jika tidak ada backup database:
# - Build image dari GHCR
# - Start service dengan database kosong
# - Data akan terisi ulang dari scraping
```
## Checklist Pencegahan
- [ ] Cron job backup database berjalan
- [ ] Backup `.env` disimpan di luar VPS (password manager)
- [ ] TLS certificates backup disimpan di luar VPS
- [ ] GitHub secrets terdaftar (tidak hanya diingat)
- [ ] Docker images bisa di-rebuild dari CI (GHCR sebagai source of truth)
- [ ] Tailscale admin access via multiple accounts
-248
View File
@@ -1,248 +0,0 @@
# CI/CD Pipeline
Dokumentasi pipeline CI/CD untuk `asepharyana-hub`. Terdiri dari 5 GitHub Actions workflow yang saling terhubung.
## Workflow Overview
```
┌─────────────┐
│ Lint │ (PR/push → Biome)
└──────┬──────┘
│
Push ke main ─────┼────── repository_dispatch
│
┌──────▼──────────────────┐
│ docker-build-push.yml │
│ │
│ Phase 1: Detect │
│ Phase 2: Build & Push │
│ Phase 3: Update │
│ manifests │
└──────┬──────────────────┘
│ workflow_run
┌──────▼──────────────┐
│ deploy-docker.yml │
│ SSH → VPS │
│ Pull → Restart │
└─────────────────────┘
repository_dispatch ──► update-submodule.yml
(dari submodule) (update pointer → commit)
│
▼
docker-build-push.yml
(triggered by push)
```
## Workflow Detail
### 1. Lint (`lint.yml`)
**Trigger:** PR/push ke `main` yang mengubah `*.json`, `*.js`, `biome.json`
**Aksi:**
- Checkout repo dengan submodules
- Setup Bun
- `bun install --frozen-lockfile`
- `bun run ci` (Biome CI mode)
**Permissions:** read-only
### 2. Build and Push Docker Images (`docker-build-push.yml`)
**Trigger:**
- Push ke `main` yang mengubah `apps/**`, `infra/**`, atau file workflow
- `repository_dispatch` tipe `submodule-updated`
- `workflow_dispatch` (manual)
**Concurrency:** Satu workflow per branch (cancel-in-progress=false)
#### Phase 1: Detect Changes
Job `changes` mendeteksi service mana yang perlu di-build:
- **Push event:** `git diff --name-only` antara `before` dan `after` SHA
- **repository_dispatch:** Parse payload `{service, sha}` dan validasi
- **workflow_dispatch:** Build semua service
Output format matrix:
```json
[{"id":"scraper-api","target":"docker-scraper","path":"apps/scraper"}]
```
#### Phase 2: Build & Push (Matrix)
Job `build` berjalan paralel per service (matrix strategy):
1. Checkout repo + sync submodule
2. Jika `repository_dispatch`, checkout submodule ke SHA tertentu
3. Login ke GHCR
4. Setup Docker Buildx
5. Build & push dengan tag:
- `ghcr.io/asepharyana/asepharyana-hub/<service>:latest`
- `ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<shortsha>`
6. Build cache: registry-based (`:<service>:buildcache`)
#### Phase 3: Update Manifests
Job `update-manifest`:
1. Update image tag di compose file (`infra/compose/<service>.yml`)
2. Jika `repository_dispatch`, update submodule pointer
3. Commit dengan message `chore: update manifests and submodules [skip ci]`
4. Push dengan retry (3 attempts, rebase jika conflict)
### 3. Deploy Docker to VPS (`deploy-docker.yml`)
**Trigger:**
- `workflow_run` setelah `docker-build-push.yml` selesai
- Push ke `main` yang mengubah `infra/**`
- `workflow_dispatch` (manual)
**Concurrency:** Satu deployment dalam satu waktu (`group: deploy-vps`)
**Aksi di VPS (via SSH):**
```
1. Setup SSH multiplexing
2. SCP .env dari GitHub secret ke VPS
3. Docker login ke GHCR
4. Git sync (fetch + reset --hard)
5. Detect changed files:
├─ Compose stack changes → selective container update
├─ Traefik dynamic config → SIGHUP
└─ Other infra → full deploy
6. Pull images (retry 3x)
7. Remove stale containers
8. Up services
9. SIGHUP Traefik jika perlu
```
### 4. Security Scan (`security.yml`)
**Trigger:**
- PR ke `main`
- Jadwal: Setiap Senin (`0 6 * * 1`)
**Aksi:**
- Checkout dengan fetch-depth 2
- CodeQL init untuk Rust
- `cargo build` di `apps/scraper`
- CodeQL analyze
### 5. Update Submodule Pointer (`update-submodule.yml`)
**Trigger:** `repository_dispatch` tipe `submodule-updated`
**Aksi:**
1. Validasi payload (`service`, `sha`)
2. Map service ke submodule path (e.g., `scraper-api` → `apps/scraper`)
3. Update submodule ke SHA yang diberikan
4. Commit sebagai `monrepo-bot` dengan message:
`chore: update <service> to <shortsha>`
5. Push dengan retry (3 attempts)
## Flow Submodule Update
Flow lengkap ketika code berubah di submodule repo:
```
1. Developer push ke asepharyana-hub-scraper
2. GitHub Action di scraper repo kirim repository_dispatch
ke asepharyana-hub
3. update-submodule.yml terima dispatch, update pointer
4. Commit masuk ke hub repo main
5. Commit ini trigger docker-build-push.yml
(push ke main dengan path apps/scraper/**)
6. Build image baru, update compose file
7. Deploy ke VPS
```
## Secrets yang Diperlukan
| Secret | Workflow | Deskripsi |
|--------|----------|-----------|
| `SSH_PRIVATE_KEY` | deploy-docker | SSH key untuk akses VPS |
| `VPS_HOST` | deploy-docker | IP VPS (`45.127.35.244`) |
| `VPS_USER` | deploy-docker | User SSH (`root`) |
| `VPS_TARGET_DIR` | deploy-docker | Dir di VPS (`/root/asepharyana-hub`) |
| `ENV_FILE_PRODUCTION` | deploy-docker | Full `.env` production |
## Menambahkan Service Baru ke Pipeline
Untuk menambahkan service baru, update:
### `docker-build-push.yml`
1. **Phase 1 — `changes` job:** Tambah detection logic untuk service baru:
```yaml
echo "new-service=$(changed '^(apps/new-service(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/new-service\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
```
2. **Phase 1 — `repository_dispatch`:** Tambah case:
```yaml
case "$SERVICE" in
scraper-api|new-service) ;;
```
3. **Phase 1 — `set-matrix`:** Tambah service:
```bash
if [ "${{ ...['new-service'] == 'true' ... }}" == "true" ]; then add_service "new-service" "docker-new-service" "apps/new-service"; fi
```
4. **Phase 2 — `meta` step:** Tambah mapping Dockerfile:
```bash
"new-service") echo "dockerfile=infra/docker/new-service.Dockerfile" >> $GITHUB_OUTPUT ;;
```
5. **Phase 3 — `update-manifest`:** Tambah mapping:
```bash
SERVICES["new-service"]="new-service.yml"
PATHS["new-service"]="apps/new-service"
```
### `deploy-docker.yml`
Tambah compose file ke `ALL_COMPOSE_FILES`:
```bash
ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/nats.yml infra/compose/dapr.yml infra/compose/new-service.yml"
```
## Rollback
### Rollback Image
```bash
# Cari SHA tag sebelumnya di GHCR packages
# Update compose file ke tag tersebut
sed -i 's|sha-badcommit|sha-goodcommit|g' infra/compose/scraper.yml
git commit -am "fix: rollback scraper-api to sha-goodcommit"
git push
```
### Rollback via Git Revert
```bash
git revert HEAD
git push origin main
# Pipeline otomatis build dan deploy
```
## Monitoring Pipeline
```bash
# Cek status workflow terbaru
gh run list --limit 5
# Lihat log workflow tertentu
gh run view <run-id> --log
# Trigger workflow manual
gh workflow run deploy-docker.yml
```
-286
View File
@@ -1,286 +0,0 @@
# NATS + JetStream Guide
Dokumentasi konfigurasi, penggunaan, dan troubleshooting NATS di infrastruktur `asepharyana-hub`.
## Arsitektur
NATS berjalan di container `nats` dengan JetStream diaktifkan (`-js`). Data persistent disimpan di volume Docker `nats_data`.
```
Service ──► NATS (port 4222) ──► JetStream (disk)
│
├─ Monitoring HTTP: port 8222
└─ Client connections: port 4222
```
### Hubungan dengan Dapr
Saat ini Dapr pub/sub menggunakan **Redis** (`pubsub.redis`), bukan NATS. NATS berfungsi sebagai message broker independen untuk:
- Event streaming antar service
- Persistent job queues
- Pub/sub untuk service yang tidak menggunakan Dapr
Jika ingin Dapr menggunakan NATS sebagai backend pub/sub, ganti komponen `pubsub.yaml`:
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.nats
version: v1
metadata:
- name: natsURL
value: nats://nats:4222
```
## Konfigurasi Compose
File: `infra/compose/nats.yml`
```yaml
services:
nats:
container_name: nats
image: nats:latest
restart: always
networks:
app-shared-net:
aliases:
- nats
ports:
- '4222:4222' # client connections
- '8222:8222' # HTTP monitor
command:
- '-js' # enable JetStream
- '-sd'
- '/data' # storage directory
volumes:
- nats_data:/data
```
## CLI Tools
### Install NATS CLI
```bash
# Linux
curl -sf https://bin.nats.dev/nats | sh
sudo mv nats /usr/local/bin/
# Atau via package manager
# brew install nats-io/nats-tools/nats (macOS)
```
### Koneksi ke NATS
```bash
# Dari host (port 4222 ter-expose)
nats context save hub --server nats://localhost:4222 --description "Hub Production"
nats context select hub
# Test koneksi
nats server check
nats server info
```
### Manage Streams (JetStream)
```bash
# List semua stream
nats stream list
# Lihat detail stream
nats stream info <stream-name>
# Buat stream
nats stream add <stream-name> \
--subjects "hub.>" \
--storage file \
--max-msgs 1000000 \
--max-bytes 1G \
--retention limits
# Hapus stream
nats stream rm <stream-name>
# Purge (hapus semua message, retain stream)
nats stream purge <stream-name>
```
### Pub/Sub
```bash
# Subscribe ke subject
nats sub "hub.>"
nats sub "hub.image.cached"
# Publish message
nats pub "hub.test" '{"message": "hello"}'
nats pub "hub.image.cached" '{"original_url": "https://example.com/img.jpg", "cdn_url": "https://cdn.example.com/img.jpg"}'
# Request-reply
nats request "hub.service.do" '{"task": "process"}'
```
### Monitoring via HTTP API
```bash
# Server info
curl http://localhost:8222/
# JetStream info
curl http://localhost:8222/jszetstream
# Stream detail
curl http://localhost:8222/jszetstream?stream=<stream-name>
# Consumer info
curl http://localhost:8222/jszetstream?stream=<stream-name>&consumer=<consumer-name>
# Server stats
curl http://localhost:8222/varz
# Connections
curl http://localhost:8222/connz
```
## Event Topics Convention
Semua topik menggunakan prefix `hub.`:
| Subject | Payload | Deskripsi |
|---------|---------|-----------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
| `hub.test` | Any | Testing |
### Wildcard Subjects
NATS mendukung wildcard:
- `hub.>` — semua event hub (multi-level)
- `hub.image.*` — semua event image (single-level)
- `hub.*.done` — semua event yang selesai (single-level)
## JetStream Configuration
### Storage
Data JetStream disimpan di volume Docker `nats_data`.
Lokasi di VPS:
```bash
docker volume inspect nats_data
# atau
ls -la /var/lib/docker/volumes/nats_data/_data/
```
### Memory & Limits
NATS tidak memiliki konfigurasi limit memori default. Untuk production, pertimbangkan:
```yaml
command:
- '-js'
- '-sd'
- '/data'
- '--max_pending_size=64MB'
- '--max_payload=1MB'
```
Atau gunakan NATS configuration file:
```yaml
# nats-server.conf
jetstream:
max_memory_store: 256MB
max_file_store: 10GB
```
## Troubleshooting
### Stream data tidak muncul
```bash
# 1. Cek koneksi NATS
nats server check
# 2. Cek apakah JetStream aktif
curl http://localhost:8222/jszetstream
# 3. Cek stream dan message count
nats stream list
# 4. Subscribe langsung untuk test
nats sub ">"
```
### NATS tidak bisa start
```bash
# Cek log
docker logs nats
# Cek apakah port 4222 sudah dipakai
ss -tlnp | grep 4222
# Cek volume data korup
docker run --rm -v nats_data:/data alpine ls -la /data
# Restart
docker compose -f infra/compose/nats.yml up -d --force-recreate
```
### Disk JetStream penuh
```bash
# Cek ukuran volume
docker system df -v | grep nats_data
# Purge stream jika perlu
nats stream purge <stream-name>
# Atau hapus volume (data hilang!)
docker compose -f infra/compose/nats.yml down
docker volume rm nats_data
docker compose -f infra/compose/nats.yml up -d
```
### Slow consumer
```bash
# Cek consumer lag
nats stream info <stream-name>
# Lihat fields: "Pending" dan "Acknowledgment"
# Lihat stats server
curl http://localhost:8222/varz | jq '.slow_consumers'
```
## Migration: Redis Pub/Sub ke NATS
Jika ingin migrasi dari Dapr pub/sub Redis ke NATS:
1. Buat stream NATS untuk topik `hub.>`
2. Update `infra/dapr/components/pubsub.yaml` dari `pubsub.redis` ke `pubsub.nats`
3. Deploy ulang semua service (Dapr sidecar akan reconnect)
4. Verifikasi event flow
```yaml
# infra/dapr/components/pubsub.yaml (setelah migrasi)
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.nats
version: v1
metadata:
- name: natsURL
value: nats://nats:4222
```
-60
View File
@@ -1,60 +0,0 @@
# Tools — Document Scanner & Media Processing Hub
Self-hosted, no-install document scanner dan media processing tools yang jalan di browser. Alternatif dari CamScanner, ilovepdf, compressjpeg — tanpa upload ke pihak ketiga.
## Visi
Satu platform dengan tools manipulasi file yang **beneran dipake orang setiap hari**. Semua proses di backend Rust — cepat, hemat memory, ga perlu install software.
## Fitur Utama
### Phase 1 — Document Scanner (Prioritas)
- Foto dokumen pake HP → auto-detect tepi → lurusin (perspective correction)
- Enhance: iluminasi merata, contrast, sharpen, B&W
- OCR → searchable PDF (teks bisa di-copy, dicari)
- Batch: multi-page → satu PDF
- Fallback crop manual (kalau auto-detect gagal)
### Phase 2 — Image Tools
- Compress JPEG/PNG/WebP (lossy + lossless, atur kualitas %)
- Resize batch (atur dimensi, semua foto disamain)
- Convert format (HEIC→JPEG, PNG→WebP, SVG→PNG)
- Remove background (ONNX model, Rust runtime)
### Phase 3 — PDF Tools
- Merge PDF (gabung file)
- Split PDF (ekstrak halaman tertentu)
- Images→PDF (kumpulan foto jadi 1 file)
- PDF→Images (tiap halaman jadi gambar)
- PDF compress (turunkin kualitas embedded images)
### Phase 4 — Video/Audio Tools
- Compress video (bitrate + resolusi)
- Extract audio (MP4→MP3)
- Trim/crop
- GIF maker
- Audio convert + trim
## Target User
Orang yang:
- Punya HP/PC, paham teknologi dasar (buka browser, upload file)
- Butuh scan dokumen tanpa install aplikasi
- Butuh kompres file buat kirim WA/email
- Butuh manipulasi PDF sesekali
- Peduli privasi — ga mau upload file ke server pihak ketiga
## Prinsip Desain
1. **Satu task selesai dalam <5 detik** — ga ada loading lama
2. **Drag & drop + preview** — liat hasil sebelum download
3. **Progress realtime** via WebSocket — tau lagi di tahap mana
4. **Batch processing** — banyak file, satu klik
5. **Privasi first** — file otomatis dihapus setelah 1 jam
6. **WASM fallback** — tools ringan jalan di client (tanpa upload)
## Domain & Branding
- **Domain**: `tools.asepharyana.my.id` | `tools.asepharyana.web.id`
- **Design**: Twilight Terminal theme (sama kaya portfolio), konsisten visual
- **Dashboard**: Link dari hub dashboard → tools stats (total files processed, storage used)
-451
View File
@@ -1,451 +0,0 @@
# Architecture
## System Overview
```
┌────────────────────────────────────────────────────────────────┐
│ BROWSER │
│ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │
│ │ Upload │ │ Camera │ │ Preview + Download │ │
│ │ (drag/drop)│ │ (PWA) │ │ (streaming) │ │
│ └─────┬──────┘ └─────┬──────┘ └───────────┬────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ WebSocket (progress: processing/step/percentage) │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────┬───────────────────────────────────┘
│ HTTPS / WSS
▼
┌────────────────────────────────────────────────────────────────┐
│ TRAEFIK (tools.asepharyana.my.id) │
│ Middleware chain: secure-headers → compress → rate-limit │
└──────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ tools-app (Next.js 16 / TypeScript) │
│ │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Pages/Routes │ │ API Routes │ │
│ │ / → home │ │ POST /api/upload ──▶ file │
│ │ /scan → scanner │ │ GET /api/job/:id ─▶ status │
│ │ /image → image │ │ WS /api/job/:id/ws ─▶ progress │
│ │ /pdf → pdf tools │ │ GET /api/download/:id ─▶ file │
│ └──────────────────┘ └─────────────────┘ │
│ │
│ Upload validation: MIME type, size limit (50MB), virus scan │
│ Temp storage bridge ke worker via HTTP/NATS │
└──────────────────┬────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌────────────────────────────────────────────────────────────────┐
│ API GATEWAY (Rust / Axum) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
│ │ Upload │ │ Job Manager │ │ Download │ │
│ │ (streaming │ │ (CRUD job │ │ (stream file, │ │
│ │ chunked) │ │ status) │ │ auto-delete) │ │
│ └──────┬───────┘ └──────┬───────┘ └────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ NATS JetStream │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ jobs │ │ jobs │ │ jobs │ │ │
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ progress │ │ progress │ │ progress │ │ │
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Cache (Redis) │ │
│ │ - Job metadata (status, progress, timestamps) │ │
│ │ - Rate limiting (sliding window per IP/tool) │ │
│ │ - Result metadata (file path, size, type) │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────┬────────────────────────────────────────────┘
│ consume NATS queue
▼
┌────────────────────────────────────────────────────────────────┐
│ WORKER POOL (Rust / Tokio + Rayon) │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ Scan Worker │ │ Image Worker │ │ PDF Worker │ │
│ │ ×4 instances │ │ ×2 instances │ │ ×2 instances│ │
│ │ │ │ │ │ │ │
│ │ 1. Load image │ │ 1. Load image │ │ 1. Load PDF │ │
│ │ 2. Edge detect │ │ 2. Compress │ │ 2. Merge/ │ │
│ │ 3. Warp │ │ /resize/ │ │ split │ │
│ │ 4. Enhance │ │ convert │ │ 3. Save │ │
│ │ 5. OCR │ │ 3. Save │ │ 4. Update │ │
│ │ 6. Gen PDF │ │ 4. Update │ │ job │ │
│ │ 7. Update job │ │ job status │ │ status │ │
│ │ └───────────────┘ └─────────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Temp Storage (filesystem volume / S3-compatible) │ │
│ │ Auto-cleanup: job TTL 1 jam, NATS cron tiap 10m │ │
│ └────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
```
## Component Diagram
```
┌────────────────────────────────────────────┐
│ apps/tools │
│ │
│ ├── frontend/ │
│ │ ├── pages/ ← Next.js pages │
│ │ ├── components/ ← React components │
│ │ ├── lib/ ← utilities │
│ │ └── public/ ← static assets │
│ │ │
│ ├── backend/ ← Rust workspace │
│ │ ├── gateway/ ← Axum API server │
│ │ ├── workers/ ← Processing workers │
│ │ │ ├── scanner/ ← Document scanner │
│ │ │ ├── image/ ← Image tools │
│ │ │ └── pdf/ ← PDF tools │
│ │ └── common/ ← Shared libs │
│ │ │
│ └── Dockerfile │
└────────────────────────────────────────────┘
```
## Data Flow (Document Scanner — Flow Lengkap)
```
1. User buka tools.asepharyana.my.id/scan
2. Upload foto via drag-drop atau kamera HP (PWA)
3. Next.js route handler menerima file
├─ Validasi: MIME type (image/*), max 50MB, virus header scan
└─ Upload chunked ke Gateway internal (HTTP POST)
4. Gateway menerima stream:
├─ Simpan ke temp storage
├─ Buat job record di Redis: {id, tool: "scan", status: "queued", progress: 0}
└─ Publish ke NATS: tools.scan.jobs {job_id, file_path, options}
5. Scan Worker consume dari NATS:
├─ Update Redis: status = "processing", progress = 10
├─ Load image (image-rs)
├─ Pipeline (detail di pipeline.md):
│ 1. Edge detection ──▶ progress 25
│ 2. Perspective warp ──▶ progress 40
│ 3. Shadow removal ──▶ progress 55
│ 4. Binarization ──▶ progress 70
│ 5. Contrast/sharpen ──▶ progress 80
│ 6. OCR ──▶ progress 90
│ 7. Generate PDF ──▶ progress 95
├─ Simpan file hasil ke temp storage
├─ Update Redis: status = "completed", progress = 100, result_path, ocr_text
└─ Publish ke NATS: tools.scan.progress {job_id, status, progress}
6. WebSocket handler di Gateway:
├─ Subscribe NATS topics tools.scan.progress
├─ Forward ke browser user (per-job-id filter)
└─ Browser update progress bar + preview
7. User download PDF:
├─ GET /api/download/:job_id
├─ Gateway stream file dari temp storage
└─ Browser save file
```
## Tech Stack
### Frontend (Next.js + TypeScript)
| Library | Fungsi |
|---------|--------|
| Next.js 16 | App router, API routes |
| shadcn/ui + Tailwind v4 | UI components |
| Framer Motion | Animasi progress, transisi |
| Canvas API | Preview crop manual, image manipulation client-side |
| WebSocket API | Real-time progress |
### Backend (Rust)
| Crate | Fungsi |
|-------|--------|
| `axum` | HTTP server (Gateway) |
| `tokio` | Async runtime |
| `image` | Image I/O, resize, convert, compress |
| `imageproc` | Edge detection, contour, thresholding |
| `lopdf` | PDF generation, merge, split, compress |
| `leptess` | Tesseract OCR binding |
| `ort` | ONNX Runtime (background removal) |
| `async-nats` | NATS JetStream client |
| `deadpool-redis` | Redis connection pool |
| `redis` | Redis async client |
| `rayon` | Parallel processing (batch, pixel ops) |
| `serde` | Serialization |
| `tracing` + `opentelemetry` | Observability |
| `uuid` | Job ID generation |
### Infrastructure
| Komponen | Fungsi |
|----------|--------|
| NATS JetStream | Job queue, progress pub/sub, scheduler |
| Redis | Job metadata, rate limiting, cache |
| PostgreSQL | Opsional — audit log, usage statistics |
| Tesseract | OCR engine (data files di Docker image) |
| Prometheus | Metrics (jobs/min, queue depth, latency per stage) |
## Job Queue (NATS Streams & Consumers)
### Streams
```
tools-scan-jobs → 1 stream, mirror to all scan workers
tools-image-jobs → 1 stream, mirror to all image workers
tools-pdf-jobs → 1 stream, mirror to all pdf workers
tools-progress → 1 stream, all progress events (key-value by job_id)
tools-scheduler → 1 stream, cron events
```
### Subjects
```
tools.scan.jobs.{job_id} → job submission
tools.scan.progress.{job_id} → progress update (fan-out ke Gateway)
tools.image.jobs.{job_id} → job submission
tools.image.progress.{job_id} → progress update
tools.pdf.jobs.{job_id} → job submission
tools.pdf.progress.{job_id} → progress update
tools.scheduler.cleanup → cleanup expired files (every 10 min)
```
## Redis Schema
```
job:{id} → Hash {status, tool, progress, file_path, result_path, ocr_text, created_at, ttl}
rate_limit:{ip}:{tool} → Sorted Set (sliding window)
file_meta:{hash} → String {original_name, size, mime}
```
## Metrics (Prometheus)
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `tools_jobs_total` | Counter | `tool`, `status` | Total jobs processed |
| `tools_jobs_in_flight` | Gauge | `tool` | Currently processing jobs |
| `tools_queue_depth` | Gauge | `tool` | NATS queue depth |
| `tools_processing_duration` | Histogram | `tool`, `stage` | Duration per stage |
| `tools_file_size_bytes` | Histogram | `tool` | Upload file size distribution |
| `tools_rate_limit_hits` | Counter | `tool` | Rate limit violations |
## Directory Structure
```
apps/tools/
├── frontend/
│ ├── src/
│ │ ├── app/
│ │ │ ├── page.tsx # Landing page
│ │ │ ├── scan/
│ │ │ │ ├── page.tsx # Scanner page
│ │ │ │ └── result/[id]/
│ │ │ │ └── page.tsx # Result page
│ │ │ ├── image/
│ │ │ │ ├── compress/page.tsx
│ │ │ │ ├── resize/page.tsx
│ │ │ │ ├── convert/page.tsx
│ │ │ │ └── remove-bg/page.tsx
│ │ │ ├── pdf/
│ │ │ │ ├── merge/page.tsx
│ │ │ │ ├── split/page.tsx
│ │ │ │ ├── images-to-pdf/page.tsx
│ │ │ │ └── compress/page.tsx
│ │ │ ├── api/
│ │ │ │ ├── upload/route.ts
│ │ │ │ ├── job/[id]/route.ts
│ │ │ │ │ └── ws/route.ts
│ │ │ │ └── download/[id]/route.ts
│ │ │ ├── layout.tsx
│ │ │ └── globals.css
│ │ ├── components/
│ │ │ ├── upload-zone.tsx # Drag & drop area
│ │ │ ├── progress-bar.tsx # WebSocket-connected progress
│ │ │ ├── preview.tsx # Before/after preview
│ │ │ ├── crop-editor.tsx # Manual corner adjustment
│ │ │ ├── tool-layout.tsx # Consistent tool page layout
│ │ │ └── camera-capture.tsx # PWA camera interface
│ │ ├── hooks/
│ │ │ ├── use-job-status.ts # WebSocket connection
│ │ │ ├── use-upload.ts # Upload with progress
│ │ │ └── use-camera.ts # Camera access
│ │ └── lib/
│ │ ├── utils.ts
│ │ └── types.ts
│ ├── next.config.ts
│ ├── package.json
│ └── tsconfig.json
│
├── backend/
│ ├── Cargo.toml
│ ├── gateway/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── routes/
│ │ │ ├── mod.rs
│ │ │ ├── upload.rs
│ │ │ ├── job.rs
│ │ │ ├── download.rs
│ │ │ └── ws.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── publisher.rs
│ │ ├── redis/
│ │ │ ├── mod.rs
│ │ │ ├── job.rs
│ │ │ └── ratelimit.rs
│ │ ├── metrics.rs
│ │ └── config.rs
│ │
│ ├── workers/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── scanner/
│ │ │ ├── mod.rs
│ │ │ ├── pipeline.rs
│ │ │ ├── edge.rs # Edge detection
│ │ │ ├── warp.rs # Perspective correction
│ │ │ ├── enhance.rs # Shadow removal, B&W, contrast
│ │ │ ├── ocr.rs # Tesseract wrapper
│ │ │ └── pdf.rs # Generate searchable PDF
│ │ ├── image/
│ │ │ ├── mod.rs
│ │ │ ├── compress.rs
│ │ │ ├── resize.rs
│ │ │ ├── convert.rs
│ │ │ └── remove_bg.rs
│ │ ├── pdf/
│ │ │ ├── mod.rs
│ │ │ ├── merge.rs
│ │ │ ├── split.rs
│ │ │ ├── extract.rs
│ │ │ └── compress.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── consumer.rs
│ │ └── config.rs
│ │
│ └── common/
│ ├── Cargo.toml
│ └── src/
│ ├── lib.rs
│ ├── types.rs # Shared types (JobStatus, Job, etc.)
│ ├── error.rs # Error types
│ └── nats.rs # NATS subject constants
│
├── Dockerfile
├── compose.yml # Local dev compose
└── README.md
```
## API Design
### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/upload` | Upload file, create job |
| `GET` | `/api/job/:id` | Get job status + result metadata |
| `WS` | `/api/job/:id/ws` | WebSocket — realtime progress |
| `GET` | `/api/download/:id` | Download result file |
| `DELETE` | `/api/job/:id` | Cancel job, delete files |
| `GET` | `/health` | Health check |
### Upload Request
```
POST /api/upload
Content-Type: multipart/form-data
{
file: <binary>,
tool: "scan" | "image-compress" | "image-resize" | "image-convert" | "remove-bg" |
"pdf-merge" | "pdf-split" | "images-to-pdf" | "pdf-compress",
options?: { // tool-specific options
quality?: 80, // compress quality
width?: 1920, // resize width
format?: "webp", // convert format
pages?: "1,3-5", // PDF split pages
dpi?: 300, // scan DPI
enhance?: true, // scan auto-enhance
ocr?: true // scan OCR
}
}
```
### Response (202 Accepted)
```json
{
"job_id": "uuid",
"status": "queued",
"tool": "scan",
"ws_url": "/api/job/uuid/ws",
"created_at": "2026-07-24T10:00:00Z",
"estimated_seconds": 5
}
```
### WebSocket Messages
```json
// Server → Client
{
"type": "progress",
"job_id": "uuid",
"status": "processing",
"progress": 45,
"stage": "warp",
"message": "Meluruskan perspektif dokumen..."
}
{
"type": "complete",
"job_id": "uuid",
"status": "completed",
"progress": 100,
"result": {
"download_url": "/api/download/uuid",
"file_name": "scan_20260724.pdf",
"file_size": 1245678,
"pages": 1,
"ocr_text": "Nama: Asep...",
"preview_url": "/api/job/uuid/preview"
}
}
{
"type": "error",
"job_id": "uuid",
"status": "failed",
"error": "Edge detection failed: cannot find document boundary"
}
```
## Integration with Existing Portfolio
| Area | Detail |
|------|--------|
| **Domain** | `tools.asepharyana.my.id` — tambah entry di `infra/traefik/dynamic/apps.yaml` |
| **Dashboard** | Link ke tools stats di dashboard hub yang sudah ada |
| **Docker Compose** | `infra/compose/tools.yml` — pola sama kaya `hub.yml` |
| **CI/CD** | Tambah service `tools` di `docker-build-push.yml` |
| **Style** | Ulang Twilight Terminal theme dari hub, konsisten visual branding |
| **Monitoring** | Reuse existing Prometheus + Grafana, tambah metrics tools |
File diff suppressed because it is too large Load Diff
-427
View File
@@ -1,427 +0,0 @@
# Infrastructure & Deployment
## Docker Image Architecture
Project ini punya **satu Docker image** dengan multi-stage build. Backend Rust + Tesseract + ONNX model plus frontend Next.js.
### Dockerfile Structure
```dockerfile
# ============================================================
# Stage 1: Build Rust Backend
# ============================================================
FROM rust:1.85-slim-bookworm AS chef
RUN cargo install cargo-chef
WORKDIR /app
FROM chef AS planner
COPY backend/ .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY backend/ .
RUN cargo build --release --bin gateway --bin workers
# ============================================================
# Stage 2: Build Next.js Frontend
# ============================================================
FROM oven/bun:1.3 AS frontend-builder
WORKDIR /app
COPY frontend/package.json frontend/bun.lock ./
RUN bun install --frozen-lockfile
COPY frontend/ .
RUN bun run build
# ============================================================
# Stage 3: Production Runtime
# ============================================================
FROM debian:bookworm-slim AS runtime
# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
tesseract-ocr \
tesseract-ocr-eng \
tesseract-ocr-ind \
ca-certificates \
fonts-dejavu-core \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy Rust binaries
COPY --from=builder /app/target/release/gateway /app/gateway
COPY --from=builder /app/target/release/workers /app/workers
# Copy Next.js build
COPY --from=frontend-builder /app/.next /app/.next
COPY --from=frontend-builder /app/public /app/public
COPY --from=frontend-builder /app/package.json /app/package.json
COPY --from=frontend-builder /app/node_modules /app/node_modules
# Copy ONNX model (for background removal)
COPY models/ /app/models/
# Create temp storage directory
RUN mkdir -p /data/tools && chmod 1777 /data/tools
# Environment
ENV TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
ENV TOOLS_STORAGE_PATH=/data/tools
ENV TOOLS_GATEWAY_PORT=3001
ENV TOOLS_WORKER_CONCURRENCY=4
ENV RUST_LOG=info
# Expose port
EXPOSE 3001
# Run both gateway and workers via supervisor script
COPY scripts/entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
CMD ["/app/entrypoint.sh"]
```
### Entrypoint Script
```bash
#!/bin/bash
# Start Gateway (Axum HTTP server)
/app/gateway &
GATEWAY_PID=$!
# Start Worker(s)
/app/workers &
WORKER_PID=$!
# Handle graceful shutdown
trap "kill $GATEWAY_PID $WORKER_PID; exit 0" SIGINT SIGTERM
# Wait for either process to exit
wait -n $GATEWAY_PID $WORKER_PID
# If one exits, kill the other
kill $GATEWAY_PID $WORKER_PID 2>/dev/null
exit 1
```
### Image Size Estimates
| Component | Size |
|-----------|------|
| Rust binary (gateway) | ~8 MB |
| Rust binary (workers) | ~15 MB |
| Next.js build | ~10 MB |
| Tesseract + data | ~25 MB |
| ONNX model | ~50 MB |
| Base (Debian slim) | ~80 MB |
| **Total** | **~188 MB** |
> ONNX model opsional — bisa di-download runtime daripada di-include di image.
---
## Docker Compose
```yaml
# infra/compose/tools.yml
services:
tools:
container_name: tools
image: ghcr.io/asepharyana/asepharyana-hub/tools:sha-xxxxxxx
restart: always
networks:
app-shared-net:
aliases:
- tools
env_file:
- ../../.env
environment:
- REDIS_URL=redis://redis:6379
- NATS_URL=nats://nats:4222
- TOOLS_STORAGE_PATH=/data/tools
- TOOLS_GATEWAY_PORT=3001
- TOOLS_WORKER_CONCURRENCY=4
- RUST_LOG=info
volumes:
- tools_data:/data/tools
ports:
- "3001:3001"
depends_on:
redis:
condition: service_started
nats:
condition: service_started
volumes:
tools_data:
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Environment Variables (`../../.env`)
```bash
# Tools
TOOLS_GATEWAY_PORT=3001
TOOLS_WORKER_CONCURRENCY=4
TOOLS_STORAGE_PATH=/data/tools
TOOLS_JOB_TTL_SECONDS=3600
TOOLS_RATE_LIMIT_PER_MINUTE=30
TOOLS_MAX_FILE_SIZE_MB=50
TOOLS_OCR_LANG=eng+ind
# Infra (reuse existing)
REDIS_URL=redis://redis:6379
NATS_URL=nats://nats:4222
```
---
## CI/CD Integration
### Docker Build Workflow
Tambah service `tools` di `.github/workflows/docker-build-push.yml`:
```yaml
# Di job "changes" step "Detect changed services"
changed() {
printf '%s\n' "$CHANGED_FILES" | grep -Eq "$1" && echo true || echo false
}
echo "tools=$(changed '^(apps/tools(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/tools\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
# Di job "build" step "Set matrix"
if [ "${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then
add_service "tools" "docker-tools" "apps/tools"
fi
# Di job "build" step "Docker metadata"
case "$SVC_NAME" in
"tools") echo "dockerfile=infra/docker/tools.Dockerfile" >> $GITHUB_OUTPUT ;;
esac
# Di job "update-manifest"
SERVICES["tools"]="tools.yml"
PATHS["tools"]="apps/tools"
```
### Deploy Workflow
Tambah di `.github/workflows/deploy-docker.yml`:
```yaml
# Tidak perlu perubahan — deploy-docker.yml auto-detect compose file changes.
# Kalau compose/tools.yml berubah, service tools akan di-restart.
```
### Service Registration (update infra/traefik/dynamic/apps.yaml)
```yaml
tools:
rule: 'Host(`tools.asepharyana.my.id`) || Host(`tools.asepharyana.web.id`)'
entryPoints:
- websecure
tls: {}
middlewares:
- common-chain@file
service: tools-service
# ...di bagian services:
tools-service:
loadBalancer:
servers:
- url: 'http://tools:3001'
```
---
## Monitoring
### Prometheus Metrics
Tambahkan label Prometheus ke container tools:
```yaml
# Di compose tools.yml
labels:
- 'prometheus.io/scrape=true'
- 'prometheus.io/port=3001'
- 'prometheus.io/path=/metrics'
```
### Dashboard Integration
Tambah card di dashboard hub yang sudah ada:
```tsx
// Di dashboard hub — tambah section "Tools Usage"
// Data dari /api/dashboard → Prometheus query:
// rate(tools_jobs_total[24h]) — jobs per tool per hari
// sum(increase(tools_jobs_total[7d])) — total jobs minggu ini
// tools_jobs_in_flight — current processing
```
---
## Storage Architecture
### Temp Storage
```
/data/tools/
├── upload/ # Uploaded files
│ └── {job_id}.{ext}
├── processing/ # Intermediate files (stage-by-stage)
│ └── {job_id}/
│ ├── 00_original.png
│ ├── 01_grayscale.png
│ ├── 02_edges.png
│ ├── 03_warped.png
│ └── ...
└── output/ # Final output
└── {job_id}.pdf
```
### Cleanup Strategy
| Mekanisme | Timing |
|-----------|--------|
| NATS cron job | Setiap 10 menit |
| Scan files >1 jam | `find /data/tools -mmin +60 -delete` |
| Redis job keys >1 jam | `SCAN 0 MATCH job:*` → TTL check → DEL |
| Storage low warning | Alert via Notification Hub (future) |
---
## Resource Estimation (VPS orangevps)
### Current Usage
| Service | CPU | RAM | Disk |
|---------|-----|-----|------|
| Traefik | 0.1 | 50 MB | 10 MB |
| NATS | 0.05 | 30 MB | 10 MB |
| Redis | 0.05 | 10 MB | 5 MB |
| Dapr Placement | 0.02 | 20 MB | 5 MB |
| Scraper API | 0.1 | 30 MB | 50 MB |
| Hub | 0.05 | 120 MB | 200 MB |
| Jaeger | 0.1 | 200 MB | 500 MB |
| Prometheus | 0.1 | 150 MB | 1 GB |
| Node Exporter | 0.02 | 10 MB | 5 MB |
| OTel Collector | 0.05 | 50 MB | 10 MB |
| **Total Current** | **~0.64** | **~670 MB** | **~1.8 GB** |
### Tools Addition
| Resources | Estimate | Notes |
|-----------|----------|-------|
| CPU | +1.0 core (burst) | Pipeline processing berat di CPU. Scoring, warp, OCR semua CPU-bound. |
| RAM | +300 MB | Rust binary + image processing buffers + Tesseract + ONNX |
| Disk | +5 GB | Temp files, bisa lebih untuk batch processing. Butuh auto-cleanup ketat. |
| **Total After** | **~1.64 cores** | **~970 MB RAM** | **~6.8 GB disk** |
> **Catatan**: Kalau VPS cuma punya 1-2 cores, processing akan antri. NATS queue handle ini. Untuk production, pastikan CPU ada >2 cores.
### Scalability
```
VPS 1 core:
- Scanner: ~5-8 detik per page
- Concurrent: 1 job at a time
- Antrian: NATS queue buffer unlimited
VPS 4+ core:
- Scanner: ~2-3 detik per page
- Concurrent: 4 jobs parallel (1 per worker)
- Rayon: parallel per-page dalam batch
```
---
## Security Considerations
| Area | Mitigation |
|------|-----------|
| **Upload validation** | MIME type check (whitelist), magic bytes verification, max size 50MB |
| **Path traversal** | Job ID = UUID v4, no user-controlled filenames in storage |
| **Command injection** | No shell commands — semua processing via Rust crates, FFmpeg via crate binding |
| **Temporary files** | Auto-cleanup, random filenames, restricted permissions (0600) |
| **Rate limiting** | Redis sliding window: 30 requests/min/IP per tool, 429 response |
| **CORS** | Origin terbatas ke domain portfolio |
| **Resource exhaustion** | Max image dimension 8000px, max file count per batch 50, worker concurrency limit |
| **OCR data** | Tesseract data dari package manager, no user-trained models |
| **ONNX model** | Model dari source terpercaya, verify checksum |
---
## Rollback Strategy
1. **Image tag**: `tools:sha-<short>` immutable — tinggal update compose file ke tag sebelumnya
2. **Data**: Files auto-expire dalam 1 jam — no persistent data migration needed
3. **Traefik**: Cukup restart, TLS certs ga berubah
4. **Monitor**: Prometheus metrics akan langsung show error rate spike
---
## Development Setup (Local)
Untuk development tanpa Docker:
```bash
# Terminal 1: Redis + NATS
docker compose -f infra/compose/shared.yml -f infra/compose/nats.yml up -d
# Terminal 2: Rust workers
cd apps/tools/backend
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
cargo run --bin workers
# Terminal 3: Rust gateway
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
TOOLS_STORAGE_PATH=/tmp/tools \
cargo run --bin gateway
# Terminal 4: Next.js
cd apps/tools/frontend
bun dev --port 3002
```
### Test Pipeline Locally (tanpa NATS/Redis)
Untuk development pipeline image processing doang:
```rust
// Di workers/src/scanner/pipeline.rs — test function
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_full_pipeline() {
let pipeline = ScanPipeline::default();
let result = pipeline.process_sync(
"test_images/scan_miring.jpg",
ScanOptions { ocr: false, enhance: true }
);
assert!(result.is_ok());
assert!(result.unwrap().output_path.exists());
}
#[test]
fn test_edge_detection_variations() {
// Test dengan berbagai kondisi: kertas putih, background ramai, sudut ekstrim
for case in &["normal.jpg", "dark.jpg", "angle45.jpg", "shadow.jpg"] {
let img = image::open(format!("test_images/{}", case)).unwrap();
let corners = detect_corners_with_fallback(&img.grayscale().into_luma8());
assert!(corners.is_ok(), "Failed on: {}", case);
}
}
}
```
Test images kumpulin dari foto dokumen real di berbagai kondisi — ini penting buat tuning parameter.

Some files were not shown because too many files have changed in this diff Show More