refactor: large codebase cleanup - consolidate schemas, migrate to Drizzle ORM, extract frontend components, modernize Docker builds
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 2m22s
Build & Deploy / build-and-push (backend) (push) Failing after 3m22s
Build & Deploy / build-and-push (proxy) (push) Successful in 1m36s
Build & Deploy / deploy (push) Skipped

- Consolidate all DB schema definitions into packages/shared as single source of truth
- Migrate backend from raw SQL to Drizzle ORM across all modules
- Extract frontend inline UI into separate component files
- Refactor discord-gateway circuitBreaker into conversationState + moderationState
- Convert messageStore to Proxy singleton pattern
- Add validateBody/validateQuery middleware + Zod schemas for API endpoints
- Modernize Docker builds with multi-stage + pnpm deploy
- Migrate CI/CD from deployment to image-based pipeline
- Remove 60+ unused/dead files (~15K lines)
- Update color scheme from sky-blue to teal-cyan
- Move DB connection management to @bete/shared/database

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Developer
2026-07-27 21:54:31 +07:00
co-authored by Claude Opus 4.8
parent 63f21513bd
commit 5802d02e29
223 changed files with 11499 additions and 13350 deletions
@@ -0,0 +1,455 @@
# CI/CD Overhaul: Gitea CI + Container Registry Design
**Status:** Draft
**Last updated:** 2026-07-27
## 1. Problem Statement
The current CI/CD pipeline has multiple issues:
1. **Split across 3 CI systems**: GitHub Actions (build + deploy), GitLab CI (build only, no deploy), and `deploy.sh` (hot-deploy bind-mounts)
2. **Registry mismatch**: GitHub Actions pushes to `ghcr.io` but `docker-compose.yml` references `registry.gitlab.com` — the deploy route is unclear
3. **Hot-deploy complexity**: `deploy.sh` builds locally, tars dist files, SSH pipes, and binds into containers at runtime. Fragile and not reproducible
4. **No frontend in Docker**: Frontend is never built into an image — only hot-deployed via bind-mounts
5. **Stale Dockerfile**: `Dockerfile.proxy` builds a Rust WASM frontend that no longer exists
6. **Dockerfile.frontend is missing**: Frontend image doesn't exist at all
7. **Shared package fragility**: The previous refactor added `@bete/shared/database/init` export, but Docker images built from `master` don't have it — containers crash
## 2. Goal
Single CI/CD pipeline that:
- Builds Docker images for all 3 services (backend, discord-gateway, proxy-serving-frontend)
- Pushes them to Gitea's built-in Container Registry
- On the VPS, only pulls images and restarts containers — no more hot-deploy bind-mounts
- All 3 services built in one pipeline, deployed together atomically
## 3. Architecture
```
Developer pushes to main
┌────────────────────────────┐
│ Gitea Runner (server X) │
│ │
│ Job 1: build-and-push │
│ ├── bete-backend:latest │──────────▶ Gitea Container Registry
│ ├── bete-discord-gateway │──────────▶ git.imrnes.team/MythEclipse/GMW/
│ │ :latest │ bete-backend:{sha,latest}
│ └── bete-proxy:latest │──────────▶ bete-discord-gateway:{sha,latest}
│ │──────────▶ bete-proxy:{sha,latest}
│ Job 2: deploy (SSH) │
│ └─── SSH ke VPS ──────────┤
└────────────────────────────┘
┌────────────────────────────┐
│ VPS Production │
│ /opt/imphenbot/infra/ │
│ docker/ │
│ │
│ docker compose pull │
│ docker compose up -d │
│ docker image prune -f │
│ │
│ 3 containers: │
│ ┌────────┐ ┌──────────┐ │
│ │ proxy │ │ backend │ │
│ │ :80 │ │ :3000 │ │
│ └───┬────┘ └──────────┘ │
│ │ ┌─────────────┐ │
│ └────┤discord- │ │
│ │gateway │ │
│ └─────────────┘ │
└────────────────────────────┘
```
### 3.1 Service Images
| Image | From | Runs |
|-------|------|------|
| `bete-backend` | `Dockerfile.backend` | Express HTTP/WS on port 3000 |
| `bete-discord-gateway` | `Dockerfile.discord-gateway` | Discord client, internal only |
| `bete-proxy` | `Dockerfile.proxy` (rewritten) | Nginx serving frontend + proxying `/api` and `/ws` to backend |
### 3.2 Registry
Gitea provides a built-in container registry per repository at:
```
git.imrnes.team/MythEclipse/GMW/<image-name>:<tag>
```
Images are tagged with both `latest` and the commit SHA for traceability.
## 4. Files to Create / Modify
### 4.1 Create: `.gitea/workflows/deploy.yml`
One workflow, two jobs:
```yaml
name: Build & Deploy
on:
push:
branches: [main]
jobs:
build-and-push:
runs-on: ubuntu-latest
strategy:
matrix:
service: [backend, discord-gateway, proxy]
max-parallel: 2
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Gitea Registry
uses: docker/login-action@v3
with:
registry: ${{ vars.GITEA_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITEA_REGISTRY_TOKEN }}
- name: Build & Push
uses: docker/build-push-action@v6
with:
context: .
file: infra/docker/Dockerfile.${{ matrix.service }}
push: true
tags: |
${{ vars.GITEA_REGISTRY }}/${{ github.repository }}/bete-${{ matrix.service }}:${{ github.sha }}
${{ vars.GITEA_REGISTRY }}/${{ github.repository }}/bete-${{ matrix.service }}:latest
cache-from: type=gha
cache-to: type=gha,mode=max
deploy:
runs-on: ubuntu-latest
needs: build-and-push
if: github.ref == 'refs/heads/main'
steps:
- name: SSH & Deploy
uses: appleboy/ssh-action@v1.2.5
with:
host: ${{ secrets.VPS_HOST }}
username: ${{ secrets.VPS_USER }}
key: ${{ secrets.VPS_SSH_KEY }}
script: |
cd /opt/imphenbot/infra/docker
echo "${{ secrets.ENV_FILE }}" > .env
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
```
Note: Gitea CI uses GitHub Actions-compatible syntax (Act Runner). The above uses the standard `actions/*` actions and `docker/*` actions that work with both GitHub and Gitea. If Gitea's runner doesn't fully support `docker/build-push-action`, fallback to inline `docker build` and `docker push` commands.
Sensitive variables: `GITEA_REGISTRY_TOKEN`, `VPS_HOST`, `VPS_USER`, `VPS_SSH_KEY`, `ENV_FILE` set in Gitea repo Settings → Actions → Secrets. Non-sensitive: `GITEA_REGISTRY` as a Variable.
### 4.2 Rewrite: `Dockerfile.proxy`
Current proxy Dockerfile builds a Rust WASM frontend (stale — no longer exists in codebase). Replace with multi-stage build:
```dockerfile
# Stage 1: Build frontend (Next.js 16 static export)
FROM node:22-slim AS frontend-builder
WORKDIR /app
# Install pnpm
RUN corepack enable
# Copy dependency manifests
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
COPY packages/shared/package.json ./packages/shared/package.json
COPY services/frontend/package.json ./services/frontend/package.json
# Install dependencies
RUN pnpm install --frozen-lockfile --filter './services/frontend' --filter '@bete/shared'
# Copy source code
COPY packages/shared/ ./packages/shared/
COPY services/frontend/ ./services/frontend/
# Build Next.js static export
RUN pnpm --filter frontend run build
# Result in services/frontend/out/
# Stage 2: Nginx
FROM nginx:alpine
# Nginx config
COPY infra/docker/nginx/nginx.conf /etc/nginx/conf.d/default.conf
# Static frontend files
COPY --from=frontend-builder /app/services/frontend/out/ /usr/share/nginx/html/
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD wget -qO- http://localhost:80/ || exit 1
```
### 4.3 Modify: `Dockerfile.backend`
Add `VITE_BE_API_URL` and `VITE_BE_WS_URL` build args (already listed in GitHub Actions but not in Dockerfile):
```dockerfile
# Add to existing Dockerfile.backend — after FROM, before WORKDIR
ARG VITE_BE_API_URL
ARG VITE_BE_WS_URL
ENV VITE_BE_API_URL=${VITE_BE_API_URL}
ENV VITE_BE_WS_URL=${VITE_BE_WS_URL}
```
These build args are now consumed at build time for future-proofing even though they were previously only needed for frontend builds (which now lives in the proxy Dockerfile).
### 4.4 Modify: `Dockerfile.discord-gateway`
No structural changes needed — verify Drizzle migrations path:
```dockerfile
# COPY drizzle, line in existing Dockerfile.discord-gateway:
COPY services/discord-gateway/drizzle/ ./services/discord-gateway/drizzle/
# This should work as-is since workspace is copied at /app
```
### 4.5 Rewrite: `deploy.sh`
From hot-deploy tar-pipe SSH to lightweight SSH exec:
```bash
#!/bin/bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
INFRA_DIR="$SCRIPT_DIR/infra/docker"
: "${VPS_HOST:?required}"
: "${VPS_USER:?required}"
: "${VPS_SSH_KEY:?required}"
echo "=== Deploy to $VPS_HOST ==="
# Copy .env if it exists locally
if [ -f "$INFRA_DIR/.env" ]; then
scp -i "$VPS_SSH_KEY" "$INFRA_DIR/.env" "$VPS_USER@$VPS_HOST:/opt/imphenbot/infra/docker/.env"
fi
ssh -i "$VPS_SSH_KEY" "$VPS_USER@$VPS_HOST" << 'REMOTESCRIPT'
set -e
cd /opt/imphenbot/infra/docker
echo "=== Pulling images ==="
docker compose pull
echo "=== Restarting containers ==="
docker compose up -d --remove-orphans
echo "=== Cleaning up ==="
docker image prune -f
echo "=== Verify ==="
docker ps --filter "name=imphenbot" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
REMOTESCRIPT
echo "=== Deploy complete ==="
```
### 4.6 Rewrite: `infra/docker/docker-compose.yml`
Replace all GitLab registry image references with Gitea registry. Remove bind-mounts. Add recordings named volume.
```yaml
version: "3.8"
services:
proxy:
image: ${GITEA_REGISTRY}/${GITEA_REPO}/bete-proxy:${IMAGE_TAG:-latest}
container_name: imphenbot-proxy
restart: unless-stopped
ports:
- "127.0.0.1:8080:80"
networks:
- app-shared-net
healthcheck:
test: wget -qO- http://localhost:80/ || exit 1
interval: 30s
timeout: 3s
start_period: 10s
retries: 3
deploy:
resources:
limits:
memory: 64M
labels:
traefik.enable: "true"
traefik.http.routers.imphenbot.rule: "Host(`imphnen.asepharyana.my.id`)"
traefik.http.routers.imphenbot.entrypoints: websecure
traefik.http.routers.imphenbot.tls: "true"
traefik.http.services.imphenbot.loadbalancer.server.port: "80"
backend:
image: ${GITEA_REGISTRY}/${GITEA_REPO}/bete-backend:${IMAGE_TAG:-latest}
container_name: imphenbot-backend
restart: unless-stopped
env_file:
- .env
environment:
NODE_ENV: production
WEBSERVER_PORT: 3000
networks:
- app-shared-net
healthcheck:
test: wget -qO- http://localhost:3000/api/health || exit 1
interval: 30s
timeout: 5s
start_period: 20s
retries: 3
deploy:
resources:
limits:
memory: 256M
depends_on:
- proxy
discord-gateway:
image: ${GITEA_REGISTRY}/${GITEA_REPO}/bete-discord-gateway:${IMAGE_TAG:-latest}
container_name: imphenbot-discord-gateway
restart: unless-stopped
env_file:
- .env
environment:
NODE_ENV: production
volumes:
- recordings:/app/recordings
networks:
- app-shared-net
healthcheck:
test: sh -c "kill -0 1"
interval: 30s
timeout: 5s
start_period: 20s
retries: 3
deploy:
resources:
limits:
memory: 512M
volumes:
recordings:
networks:
app-shared-net:
external: true
```
Key changes:
- Image refs: `registry.gitlab.com/mytheclipse-group/gmw/...``${GITEA_REGISTRY}/${GITEA_REPO}/...`
- **All bind-mounts removed** (`./backend-dist`, `./gateway-dist`, `./frontend-dist`, `./shared-dist`)
- `recordings` → named volume (persists across container restarts/recreates)
- `proxy` binds to `127.0.0.1:8080` instead of host port 80 (Traefik handles external routing)
- Added `depends_on: proxy` to backend for startup ordering
### 4.7 Remove: GitHub Actions & GitLab CI files
After Gitea CI is verified working:
- Delete `.github/workflows/deploy-docker.yml` (or rename to `.github/workflows/deploy-docker.yml.disabled`)
- Delete `.gitlab-ci.yml` (or rename to `.gitlab-ci.yml.disabled`)
### 4.8 Ensure: `.gitea/workflows/` directory
The directory must exist in git. Some setups ignore `.gitea/` — verify `.gitignore` does not exclude it.
## 5. Gitea Registry Integration
### 5.1 Enable Container Registry in Gitea
In Gitea Admin Settings:
- Go to Settings → Repository → Enable "Container Registry"
- Default registry URL format: `gitea.<domain>/<owner>/<repo>`
### 5.2 Registry Token
Create a Gitea access token with `read` and `write` access to packages:
- Settings → Applications → Generate Token → `registry-token` → scope: `write:packages`
### 5.3 CI Variables
Set these in Gitea repo → Settings → Actions → Secrets:
| Name | Example Value | Notes |
|------|---------------|-------|
| `GITEA_REGISTRY_TOKEN` | `gitea_token_abc123` | Docker login password |
| `VPS_HOST` | `123.123.123.123` | VPS IP/hostname |
| `VPS_USER` | `root` | SSH user |
| `VPS_SSH_KEY` | `-----BEGIN OPENSSH PRIVATE KEY-----...` | Private key |
| `ENV_FILE` | full .env content | Written to VPS before compose |
As Variables (not secrets, visible but non-sensitive):
| Name | Example Value | Notes |
|------|---------------|-------|
| `GITEA_REGISTRY` | `git.imrnes.team` | Registry hostname — no protocol prefix |
### 5.4 VPS Setup (one-time)
```bash
# 1. Docker login to Gitea registry
docker login git.imrnes.team
# Use Gitea username + access token (with write:packages scope)
# 2. Create recordings named volume
docker volume create imphenbot_recordings
# 3. Remove old bind-mount directories (after verifying old containers stopped)
rm -rf /opt/imphenbot/infra/docker/backend-dist
rm -rf /opt/imphenbot/infra/docker/gateway-dist
rm -rf /opt/imphenbot/infra/docker/shared-dist
rm -rf /opt/imphenbot/infra/docker/frontend-dist
# 4. Ensure compose file is updated (via git pull)
cd /opt/imphenbot && git pull origin main
```
## 6. Migration Plan
### Phase 1: Prepare (this session)
1. Write `.gitea/workflows/deploy.yml`
2. Rewrite `Dockerfile.proxy` for Next.js
3. Modify `infra/docker/docker-compose.yml` for Gitea registry + named volumes
4. Rewrite `deploy.sh` to SSH-only
5. Mark old CI files as disabled (rename, not delete yet)
6. Add VITE_BE_API_URL/VITE_BE_WS_URL build args to backend Dockerfile
### Phase 2: VPS Preparation (one-time SSH)
7. User runs `docker login` to Gitea registry on VPS
8. User sets CI secrets in Gitea UI
9. User creates `imphenbot_recordings` named volume
### Phase 3: Deploy
10. Commit and push to `main`
11. Gitea CI triggers — builds 3 images, pushes to registry
12. Deploy job SSHes into VPS, pulls images, restarts containers
13. Verify with `docker ps` and health checks
### Phase 4: Cleanup
14. After all services running stably for 1-2 pushes: delete old CI files
15. Remove old Dockerfiles if no longer referenced
## 7. Rollback Plan
If something goes wrong:
1. **Quick rollback**: `docker compose up -d` with previous `IMAGE_TAG` (pin to last working SHA)
2. **Full rollback**: Revert git changes, push to `main` — Gitea CI will rebuild with old config
3. **Emergency**: SSH to VPS, use `docker compose` commands to restart specific containers
## 8. Future Considerations
- **Auto-deploy on tag**: Optionally trigger CI only on version tags (`v*`) instead of every `main` push
- **Health check notifications**: Add webhook notification on deploy failure
- **Multi-architecture builds**: Add `--platform linux/amd64,linux/arm64` for future ARM VPS migration
- **Secrets management**: Consider HashiCorp Vault or Gitea's built-in encrypted secrets for larger teams
@@ -0,0 +1,134 @@
# Refactoring Backend & Discord-Gateway Services
**Date:** 2026-07-27
**Status:** Draft
## Overview
Comprehensive refactoring of `services/backend` (4.2k lines) and `services/discord-gateway` (17.9k lines) targeting code consistency, file-size reduction, deduplication, and pattern uniformity.
## Scope
### Phase 1 — Backend Controller Consistency
**Problem:** Two competing controller patterns.
- `messages.controller.ts`, `mascot-chat.controller.ts` use convoluted `asyncHandler` inside function body (Gaya A)
- `voice.controller.ts`, `health.controller.ts` use clean `asyncHandler` decorator (Gaya B)
**Fix:** Convert all controllers to **Gaya B** (decorator pattern).
Before (Gaya A):
```ts
export function handleListMessages(req, res, next) {
return asyncHandler(async (req, res) => {
// ...
})(req, res, next);
}
```
After (Gaya B):
```ts
export const handleListMessages = asyncHandler(async (req, res) => {
// ...
});
```
**Files affected:**
- `modules/messages/messages.controller.ts`
- `modules/mascot-chat/mascot-chat.controller.ts`
### Phase 2 — Backend `response.ts` Cleanup
**Problem:** `success()`/`error()` helpers exist but are unused (except health controller).
**Fix:** Apply `success()` consistently to all API responses that are successful data returns. Remove `error()` if unused after audit.
**Files affected:** All route/service files that `res.json()` data.
### Phase 3 — Backend `ws/` Barrel
**Problem:** `ws/broadcast.ts`, `ws/redis-bridge.ts`, `ws/server.ts` — no barrel.
**Fix:** Add `ws/index.ts` barrel.
### Phase 4 — Gateway: Split `moderationPrompt.ts` (1015 lines)
**Problem:** Monolithic prompt file mixing all prompt types.
**Fix:** Split into:
- `prompts/text-analysis.ts` — Text moderation prompts
- `prompts/media-analysis.ts` — Image/video analysis prompts
- `prompts/stickers.ts` — Sticker analysis prompts
- `prompts/emojis.ts` — Custom emoji prompts
- `prompts/system.ts` — System prompt builder and shared helpers
### Phase 5 — Gateway: Split `moderationOrchestrator.ts` (955 lines)
**Problem:** Entry point that also contains inline text-only batch, media batch, and simple fallback.
**Fix:** Extract into:
- `textBatchProcessor.ts` — All text-only batching logic
- `mediaBatchProcessor.ts` — All media batching logic
- `simpleFallback.ts` — The `runSimpleTextFallback` function
### Phase 6 — Gateway: Split `mediaAnalysisClient.ts` (826 lines)
**Problem:** Cache logic (LRU + phash + DB), download logic (image/video + ffmpeg), and vision LLM in one file.
**Fix:** Extract into:
- `mediaCache.ts` — All caching layers (LRU, phash dedup, DB)
- `mediaDownloader.ts` — Image/video download, ffmpeg frame extraction
- `visionAnalyzer.ts` — Vision LLM orchestration
### Phase 7 — Gateway: Consolidate `bootstrap.ts`
**Problem:** 304-line bootstrap that embeds retention cleanup inline.
**Fix:** Extract `startRetentionCleanup` into `app/retention.ts`. Leave event registrations in bootstrap as they're inherently app-wide wiring.
### Phase 8 — Gateway: Simplify EventBroadcaster
**Problem:** `RedisEventPublisher` wrapping is thin — only adds a `publish` wrapper.
**Fix:** Merge `RedisEventPublisher` into `EventBroadcaster` as a private inner detail.
### Phase 9 — Cross-cutting: Database initialization dedup
**Problem:** Backend (`shared/database/index.ts`) and gateway (`shared/database/drizzle.ts`) have near-identical pool creation and lifecycle code.
**Fix:** Extract common pool/drizzle lifecycle into `@bete/shared`:
```ts
// packages/shared/src/database/index.ts
export function createDatabasePool(url: string, opts?: PoolOpts): Pool
export function createDrizzleClient(pool: Pool): DrizzleClient
export function closePool(pool: Pool): Promise<void>
```
Both services keep their own getDatabase/close wrappers but delegate pool creation to shared.
### Phase 10 — Gateway: Consolidate `moderationState.ts` / `conversationState.ts`
**Problem:** Two state files with overlapping concerns.
**Fix:** Audit both for overlap, merge if significant duplication found.
### Phase 11 — Gateway: Redis connection usage audit
**Problem:** Multiple independent Redis connections for EventBroadcaster and CommandHandler.
**Fix:** Both already need separate connections (Redis pub/sub limits). Document the pattern. No structural change.
## Files Changed
| Phase | Files | Type |
|-------|-------|------|
| 1 | 3 | edit |
| 2 | ~15 | edit |
| 3 | 1 | create |
| 4 | ~6 | split |
| 5 | ~4 | split |
| 6 | ~4 | split |
| 7 | 2 | split |
| 8 | 2 | refactor |
| 9 | 2 | refactor |
| 10 | 1-2 | audit+merge |
@@ -0,0 +1,37 @@
# Visual Redesign: Discord Automod Dashboard
## Design Direction
**Vibe:** "Monitoring hub" — deep, technical, trustworthy. Think security operations center meets modern dev tool.
## Palette
**Dark (primary):**
| Token | Value | Role |
|-------|-------|------|
| `--bg` | `oklch(0.09 0.015 245)` | Deeper navy canvas |
| `--card` | `oklch(0.13 0.02 245)` | Surface with subtle separation |
| `--primary` | `oklch(0.62 0.17 215)` | Teal-cyan accent (shift from sky blue) |
| `--accent` | `oklch(0.7 0.18 260)` | Electric blue-purple for secondary highlights |
| `--warn` | `oklch(0.7 0.17 75)` | Amber-gold for warnings (distinct from red) |
| `--border` | `oklch(1 0 0 / 0.06)` | Softer borders |
## Typography
- Geist Sans (body) + Geist Mono (code/data) — already loaded
- H1: `text-lg font-semibold tracking-tight`
- Card titles: `text-sm font-semibold tracking-tight`
- Labels/captions: `text-xs text-muted-foreground tracking-wide uppercase`
## Layout Changes
1. **Background**: Subtle dot-grid pattern (`radial-gradient(circle, oklch(1 0 0 / 0.03) 1px, transparent 1px)`) — monitoring station feel
2. **Sidebar**: Slightly wider (w-64), active item gets a glow bar + subtle teal tint background, connection dot with breathing animation
3. **Cards**: Hover state adds a thin teal border-top glow, softer shadow
4. **Stat cards**: Gradient background per stat type (like live-stats had), with icon in colored bubble
5. **Severity indicators**: Colored dot + label instead of just colored border
6. **Mobile nav**: Tighter spacing, active indicator as dot above icon
7. **Header**: Clean, thin bottom border glow, page title larger
## Signature Element
- **Grid background** + **teal glow** on active/interactive elements
- **Gradient accent bar** on sidebar active item (wider, glowing)