178 lines
8.2 KiB
Markdown
178 lines
8.2 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
Scraper service — a Rust/Axum backend for web scraping (anime/komik data extraction) and image proxy/CDN caching. Serves as the backend engine consumed by the `apps/solidjs` frontend.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# Development
|
|
cargo run # Start server (binds 0.0.0.0:4091)
|
|
cargo test # Run all tests
|
|
cargo clippy -- -D warnings # Lint (warnings are errors)
|
|
cargo fmt # Auto-format all source files
|
|
|
|
# Release build (full LTO, single CGU, stripped)
|
|
cargo build --release
|
|
|
|
# PM2 production
|
|
pm2 start ecosystem.config.cjs --env production # Uses target/release/scraper
|
|
```
|
|
|
|
## Architecture
|
|
|
|
### Modular MVC + Service + Repository
|
|
|
|
Setiap module mengikuti arsitektur layered yang identik:
|
|
|
|
```
|
|
Request → Router (route.rs) → Controller → Service → Repository → Parser
|
|
│
|
|
├── Redis (L1 cache)
|
|
├── SeaORM/MySQL (L2, image_cache)
|
|
└── External HTTP (alqanime.si, picser CDN)
|
|
```
|
|
|
|
### Directory Layout
|
|
|
|
```
|
|
src/
|
|
├── main.rs # Entry point: builds Application, calls run()
|
|
├── lib.rs # Public module declarations
|
|
├── app.rs # Router assembly: modules + metrics + swagger + middleware layers
|
|
├── bootstrap/mod.rs # Application::build(): tracing, Redis, browser pool, DB, AppState
|
|
├── modules/ # Feature modules (vertical slices)
|
|
│ ├── anime/ # Otakudesu anime scraper
|
|
│ ├── anime2/ # Alqanime.si anime scraper
|
|
│ ├── komik/ # Komik scraper
|
|
│ └── proxy/ # Image proxy/cache/audit endpoints
|
|
└── shared/ # Cross-cutting infrastructure
|
|
├── config/ # Lazy-static AppConfig from env vars (fail-fast at startup)
|
|
├── state/ # AppState (redis_pool, db, semaphore, event_bus)
|
|
├── database/
|
|
│ ├── traits/ # ScrapingRepository, ImageCacheRepository (async_trait)
|
|
│ ├── repositories/ # SeaOrmImageCacheRepository (impl ImageCacheRepository)
|
|
│ └── persistence/ # SeaORM entities (image_cache)
|
|
├── services/images/ # ImageCache service + apply_cached_posters helper
|
|
├── errors/ # AppError enum → axum IntoResponse (500/404 by variant)
|
|
├── observability/ # Utoipa/Swagger OpenAPI doc
|
|
├── scheduler/ # Cron jobs (daily cache cleanup at 2 AM)
|
|
├── browser/ # Headless Chrome pool for JS-rendered scraping
|
|
├── scrapers/ # Site-specific scrapers (otakudesu)
|
|
├── utils/ # Cache helper, HTTP client, scraping helpers, retry, conversions
|
|
├── middlewares/ # Logging, rate limiting
|
|
├── events/ # EventBus for repair state updates
|
|
└── types/ # ApiResponse<T>, shared entity types (HasPoster trait, Pagination)
|
|
```
|
|
|
|
### Module Structure (identik untuk setiap module)
|
|
|
|
Setiap `src/modules/<name>/`:
|
|
|
|
| File | Peran | Pola |
|
|
|---|---|---|
|
|
| `route.rs` | Daftar endpoint, mapping URL → controller | `Router<Arc<AppState>>`, tidak ada logic |
|
|
| `controller.rs` | Extract State/Path/Query/Body, panggil service | `Result<Json<T>, AppError>` |
|
|
| `service.rs` | Business logic, caching, delegasi ke repository + parser | Struct dengan repo di-inject via constructor `new(repo: XRepository)` |
|
|
| `repository.rs` | HTTP fetching, URL builders, DB queries | Struct + `impl ScrapingRepository` trait |
|
|
| `parser.rs` | HTML parsing dengan `scraper` crate | Free functions → `Result<T, AppError>`, via `spawn_blocking` |
|
|
| `schema.rs` | Validasi query/path/body params | Struct `Deserialize` + `ToSchema` |
|
|
| `types.rs` | Response structs | `Serialize` + `ToSchema`, `impl HasPoster` jika punya poster |
|
|
|
|
### Dependency Injection
|
|
|
|
Semua service menerima dependency via constructor:
|
|
|
|
```rust
|
|
// Controller creates and injects dependencies
|
|
let repo = AnimeRepository::new();
|
|
let service = AnimeService::new(repo);
|
|
service.get_anime_index(app_state).await.map(Json)
|
|
|
|
// Service stores injected repo
|
|
pub struct AnimeService {
|
|
repository: AnimeRepository,
|
|
}
|
|
impl AnimeService {
|
|
pub fn new(repository: AnimeRepository) -> Self { Self { repository } }
|
|
}
|
|
```
|
|
|
|
### Image Caching Architecture
|
|
|
|
Single unified image cache system:
|
|
|
|
1. **Trait**: `ImageCacheRepository` (`shared/database/traits/image_cache.rs`) — Redis ops, DB ops, locks, cache invalidation
|
|
2. **Impl**: `SeaOrmImageCacheRepository` (`shared/database/repositories/image_cache.rs`)
|
|
3. **Service**: `ImageCache` struct (`shared/services/images/cache.rs`) — download, MIME-verify dengan `infer`, upload ke Picser CDN, verifikasi CDN URL (10 retry dengan backoff)
|
|
4. **Concurrency**: `Semaphore` (default 5 concurrent uploads) + request coalescing via `DashMap<broadcast::Sender>`
|
|
5. **Lazy batch helper**: `cache_image_urls_batch_lazy()` — Redis batch check → DB batch check → background spawn untuk misses
|
|
|
|
### HasPoster Trait & apply_cached_posters
|
|
|
|
`HasPoster` trait di `shared/types/entities/anime.rs` memungkinkan generic poster caching:
|
|
|
|
```rust
|
|
pub trait HasPoster {
|
|
fn poster(&self) -> &str;
|
|
fn set_poster(&mut self, url: String);
|
|
}
|
|
```
|
|
|
|
Semua item type dengan field `poster` mengimplementasikan trait ini (`OngoingAnimeItem`, `KomikItem`, `FilterAnimeItem`, `Recommendation`, dll).
|
|
|
|
`apply_cached_posters()` di `shared/services/images/cache.rs` menerima `&mut [T]` where `T: HasPoster`, menggantikan pola manual ~15 baris yang sebelumnya berulang di setiap service method.
|
|
|
|
### ScrapingRepository Trait
|
|
|
|
```rust
|
|
#[async_trait]
|
|
pub trait ScrapingRepository: Send + Sync {
|
|
async fn fetch_html(&self, url: &str) -> Result<String, AppError>;
|
|
}
|
|
```
|
|
|
|
Semua module repository (`AnimeRepository`, `Anime2Repository`, `KomikRepository`, `ProxyRepository`) mengimplementasikan trait ini.
|
|
|
|
### Error Handling
|
|
|
|
`AppError` enum di `src/shared/errors/app_error.rs` — derives `thiserror::Error` dan implements `IntoResponse` (404 untuk `NotFound`, 500 untuk lainnya).
|
|
|
|
**Kontrak error per layer:**
|
|
- **Parser** → `Result<T, AppError>`
|
|
- **Repository** → `Result<T, AppError>` (via `ScrapingRepository` trait)
|
|
- **Service** → `Result<T, AppError>` (tidak ada `Result<T, String>` atau `Box<dyn Error>`)
|
|
- **Controller** → `Result<Json<T>, AppError>` (kecuali proxy yang return raw `Response`)
|
|
|
|
### Configuration
|
|
|
|
`src/shared/config/mod.rs` — global `CONFIG` lazy-static loaded from:
|
|
1. `.env` file (dotenvy)
|
|
2. `config/default.toml` / `config/{RUN_MODE}.toml`
|
|
3. Environment variables (`APP__` prefix or legacy `DATABASE_URL`/`JWT_SECRET`/`REDIS_URL`)
|
|
|
|
Panics at startup if required config is missing — intentional fail-fast design.
|
|
|
|
## Constraints
|
|
|
|
- **No suppression flags**: `#[allow(...)]`, `#[ignore]`, `@ts-ignore` are prohibited. Fix the underlying issue.
|
|
- **Lint strictness**: `unsafe_code = "forbid"`, `panic = "deny"`, `todo = "deny"`, `unimplemented = "deny"`, `unwrap_used = "warn"`, `expect_used = "warn"`
|
|
- **Minimal dependencies**: Before adding a crate, evaluate if existing deps or std can handle it.
|
|
- **Dead code**: Remove unused functions, types, modules rather than leaving them.
|
|
- **Performance**: Use `spawn_blocking` for CPU-heavy work (HTML parsing).
|
|
- **No duplicate infrastructure**: Satu trait, satu impl. Jangan membuat trait/repository duplikat seperti `ImageRepository` dan `ImageCacheRepository` yang berbeda.
|
|
- **No thin wrappers**: Hindari wrapper tipis seperti `CacheImageUseCase` yang hanya meneruskan panggilan ke service lain.
|
|
|
|
## Useful Endpoints
|
|
|
|
- `GET /docs` — Swagger UI
|
|
- `GET /api-docs/openapi.json` — OpenAPI spec
|
|
- `POST /api/proxy/image-cache` — Cache an image URL
|
|
- `POST /api/proxy/image-cache/audit` — Audit/repair cached images
|
|
- `GET /api/anime/*` — Otakudesu scraping endpoints
|
|
- `GET /api/anime2/*` — Alqanime scraping endpoints
|
|
- `GET /api/komik/*` — Komik scraping endpoints
|