asepharyana-hub-scraper/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Scraper service — a Rust/Axum backend for web scraping (anime/komik data extraction) and image proxy/CDN caching. Serves as the backend engine consumed by the apps/solidjs frontend.

Commands

# Development
cargo run                          # Start server (binds 0.0.0.0:4091)
cargo test                         # Run all tests
cargo clippy -- -D warnings        # Lint (warnings are errors)
cargo fmt                          # Auto-format all source files

# Release build (full LTO, single CGU, stripped)
cargo build --release

# PM2 production
pm2 start ecosystem.config.cjs --env production    # Uses target/release/scraper

Architecture

Modular MVC + Service + Repository

Setiap module mengikuti arsitektur layered yang identik:

Request → Router (route.rs) → Controller → Service → Repository → Parser
                                                    │
                                                    ├── Redis (L1 cache)
                                                    ├── SeaORM/MySQL (L2, image_cache)
                                                    └── External HTTP (alqanime.si, picser CDN)

Directory Layout

src/
├── main.rs                  # Entry point: builds Application, calls run()
├── lib.rs                   # Public module declarations
├── app.rs                   # Router assembly: modules + metrics + swagger + middleware layers
├── bootstrap/mod.rs         # Application::build(): tracing, Redis, browser pool, DB, AppState
├── modules/                 # Feature modules (vertical slices)
│   ├── anime/               # Otakudesu anime scraper
│   ├── anime2/              # Alqanime.si anime scraper
│   ├── komik/               # Komik scraper
│   └── proxy/               # Image proxy/cache/audit endpoints
└── shared/                  # Cross-cutting infrastructure
    ├── config/              # Lazy-static AppConfig from env vars (fail-fast at startup)
    ├── state/               # AppState (redis_pool, db, semaphore, event_bus)
    ├── database/
    │   ├── traits/          # ScrapingRepository, ImageCacheRepository (async_trait)
    │   ├── repositories/    # SeaOrmImageCacheRepository (impl ImageCacheRepository)
    │   └── persistence/     # SeaORM entities (image_cache)
    ├── services/images/     # ImageCache service + apply_cached_posters helper
    ├── errors/              # AppError enum → axum IntoResponse (500/404 by variant)
    ├── observability/       # Utoipa/Swagger OpenAPI doc
    ├── scheduler/           # Cron jobs (daily cache cleanup at 2 AM)
    ├── browser/             # Headless Chrome pool for JS-rendered scraping
    ├── scrapers/            # Site-specific scrapers (otakudesu)
    ├── utils/               # Cache helper, HTTP client, scraping helpers, retry, conversions
    ├── middlewares/         # Logging, rate limiting
    ├── events/              # EventBus for repair state updates
    └── types/               # ApiResponse<T>, shared entity types (HasPoster trait, Pagination)

Module Structure (identik untuk setiap module)

Setiap src/modules/<name>/:

File	Peran	Pola
route.rs	Daftar endpoint, mapping URL → controller	Router<Arc<AppState>>, tidak ada logic
controller.rs	Extract State/Path/Query/Body, panggil service	Result<Json<T>, AppError>
service.rs	Business logic, caching, delegasi ke repository + parser	Struct dengan repo di-inject via constructor new(repo: XRepository)
repository.rs	HTTP fetching, URL builders, DB queries	Struct + impl ScrapingRepository trait
parser.rs	HTML parsing dengan scraper crate	Free functions → Result<T, AppError>, via spawn_blocking
schema.rs	Validasi query/path/body params	Struct Deserialize + ToSchema
types.rs	Response structs	Serialize + ToSchema, impl HasPoster jika punya poster

Dependency Injection

Semua service menerima dependency via constructor:

// Controller creates and injects dependencies
let repo = AnimeRepository::new();
let service = AnimeService::new(repo);
service.get_anime_index(app_state).await.map(Json)

// Service stores injected repo
pub struct AnimeService {
    repository: AnimeRepository,
}
impl AnimeService {
    pub fn new(repository: AnimeRepository) -> Self { Self { repository } }
}

Image Caching Architecture

Single unified image cache system:

1. Trait: ImageCacheRepository (shared/database/traits/image_cache.rs) — Redis ops, DB ops, locks, cache invalidation
2. Impl: SeaOrmImageCacheRepository (shared/database/repositories/image_cache.rs)
3. Service: ImageCache struct (shared/services/images/cache.rs) — download, MIME-verify dengan infer, upload ke Picser CDN, verifikasi CDN URL (10 retry dengan backoff)
4. Concurrency: Semaphore (default 5 concurrent uploads) + request coalescing via DashMap<broadcast::Sender>
5. Lazy batch helper: cache_image_urls_batch_lazy() — Redis batch check → DB batch check → background spawn untuk misses

HasPoster Trait & apply_cached_posters

HasPoster trait di shared/types/entities/anime.rs memungkinkan generic poster caching:

pub trait HasPoster {
    fn poster(&self) -> &str;
    fn set_poster(&mut self, url: String);
}

Semua item type dengan field poster mengimplementasikan trait ini (OngoingAnimeItem, KomikItem, FilterAnimeItem, Recommendation, dll).

apply_cached_posters() di shared/services/images/cache.rs menerima &mut [T] where T: HasPoster, menggantikan pola manual ~15 baris yang sebelumnya berulang di setiap service method.

ScrapingRepository Trait

#[async_trait]
pub trait ScrapingRepository: Send + Sync {
    async fn fetch_html(&self, url: &str) -> Result<String, AppError>;
}

Semua module repository (AnimeRepository, Anime2Repository, KomikRepository, ProxyRepository) mengimplementasikan trait ini.

Error Handling

AppError enum di src/shared/errors/app_error.rs — derives thiserror::Error dan implements IntoResponse (404 untuk NotFound, 500 untuk lainnya).

Kontrak error per layer:

• Parser → Result<T, AppError>
• Repository → Result<T, AppError> (via ScrapingRepository trait)
• Service → Result<T, AppError> (tidak ada Result<T, String> atau Box<dyn Error>)
• Controller → Result<Json<T>, AppError> (kecuali proxy yang return raw Response)

Configuration

src/shared/config/mod.rs — global CONFIG lazy-static loaded from:

1. .env file (dotenvy)
2. config/default.toml / config/{RUN_MODE}.toml
3. Environment variables (APP__ prefix or legacy DATABASE_URL/JWT_SECRET/REDIS_URL)

Panics at startup if required config is missing — intentional fail-fast design.

Constraints

• No suppression flags: #[allow(...)], #[ignore], @ts-ignore are prohibited. Fix the underlying issue.
• Lint strictness: unsafe_code = "forbid", panic = "deny", todo = "deny", unimplemented = "deny", unwrap_used = "warn", expect_used = "warn"
• Minimal dependencies: Before adding a crate, evaluate if existing deps or std can handle it.
• Dead code: Remove unused functions, types, modules rather than leaving them.
• Performance: Use spawn_blocking for CPU-heavy work (HTML parsing).
• No duplicate infrastructure: Satu trait, satu impl. Jangan membuat trait/repository duplikat seperti ImageRepository dan ImageCacheRepository yang berbeda.
• No thin wrappers: Hindari wrapper tipis seperti CacheImageUseCase yang hanya meneruskan panggilan ke service lain.

Useful Endpoints

• GET /docs — Swagger UI
• GET /api-docs/openapi.json — OpenAPI spec
• POST /api/proxy/image-cache — Cache an image URL
• POST /api/proxy/image-cache/audit — Audit/repair cached images
• GET /api/anime/* — Otakudesu scraping endpoints
• GET /api/anime2/* — Alqanime scraping endpoints
• GET /api/komik/* — Komik scraping endpoints