asepharyana
8b9bdeb56a
Facebook: - Broaden URL validation to accept ALL real FB shapes: /watch/?v=, /reel/:id, /videos/:id (user vids), /video.php, /story.php, fb.watch, and no-www facebook.com. Old regex rejected /reel, /videos, fb.watch. - Add yt-dlp as the PRIMARY provider (authoritative, returns direct fbcdn progressive URLs with real title/format metadata, no junk). Only falls back to downr.org then Playwright. This kills the 'link exists but 404s' bug where Playwright harvested broken bytestart byte-range chunks alongside the one good URL. - Filter Playwright-harvested bytestart= chunk URLs + dedupe. TeraBox: - Add /proxy/terabox streaming endpoint that forwards the resolver's cookie-authenticated stream through the public API. - Rewrite the returned download URL to the public proxy (raw dlink 403s without cookies; TERABOX_PUBLIC_BASE_URL env override). - Fix: server_port env is APP_SERVER_PORT (single underscore); the OLD stale 4091 bind in tests came from APP__SERVER_PORT not mapping. flake: bump scraper-0.1.0 -> 0.1.1
Scraper API
Backend service berbasis Axum untuk scraping, image proxy/cache, dan endpoint API inti.
Stack
- Rust + Axum
- SeaORM (MySQL)
- Redis (deadpool-redis)
- Utoipa + Swagger UI
Quick Start
cargo run
Server bind ke 0.0.0.0:${PORT} dengan default port 4091.
Required Environment Variables
DATABASE_URL=mysql://asephs:hunterz@localhost:3306/sosmed
JWT_SECRET=change-me
REDIS_URL=redis://localhost:6379
Optional yang sering dipakai:
RUST_LOG=info
MINIO_ENDPOINT=
MINIO_BUCKET_NAME=
MINIO_ACCESS_KEY=
MINIO_SECRET_KEY=
Useful Endpoints
GET /docs- Swagger UIGET /api-docs/openapi.json- OpenAPI JSONGET /api/anime2/*GET /api/komik/*POST /api/proxy/image-cachePOST /api/proxy/image-cache/audit
Notes
- Database akan dicek/dibuat saat startup jika
DATABASE_URLbertipe MySQL. - Service ini juga menginisialisasi browser pool dan scheduler saat boot.
License
MIT