T

Deploy Scraper / build-and-deploy (push) Canceled after 0s

Details

fix(downloader): terabox proxy as primary link + accept threads.net URLs

- Terabox: return the working cookie-authenticated proxy URL as the
  FIRST media entry; keep the raw dlink second with an honest note
  (it 403s without cookies).
- Threads: accept threads.net URLs in the snapsave/IG validator so
  /download/threads no longer rejects them.

2026-09-07 13:40:32 +07:00

.github

ci: deploy via Attic binary cache (attic.asepharyana.my.id) instead of ssh copy

2026-08-27 20:38:36 +07:00

.hermes/plans

fix: map ScrapingError to 502 Bad Gateway instead of 500 Internal Server Error; fix downr.org Cloudflare block; fix YouTube savetube DNS failure; fix twitter/pinterest error handling

2026-09-01 11:26:29 +07:00

docs/superpowers

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

scripts

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

src

fix(downloader): terabox proxy as primary link + accept threads.net URLs

2026-09-07 13:40:32 +07:00

tests

refactor: migrate thread/async/queue/scheduler infra to mytheclipse

2026-08-30 19:40:01 +07:00

tools/auto-lint

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

.gitignore

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

AGENT.md

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

Cargo.lock

feat(downloader): add 26 downloader APIs + detect endpoint

2026-09-01 00:54:06 +07:00

Cargo.toml

feat(downloader): add 26 downloader APIs + detect endpoint

2026-09-01 00:54:06 +07:00

check_image_cache_db.sh

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

CLAUDE.md

docs: update CLAUDE.md alqanime.si -> alqanime.net

2026-07-22 13:43:11 +07:00

ecosystem.config.cjs

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

flake.lock

fix: replace dead YouTube APIs with yt-dlp subprocess; fix all JSON panic crashes; add yt-dlp helper functions

2026-09-01 12:51:52 +07:00

flake.nix

fix(downloader): real FB media for all URL shapes + public terabox proxy

2026-09-07 13:20:11 +07:00

GEMINI.md

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

package.json

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

README.md

refactor: remove EXTERNAL_BROWSERLESS_WS / browserless fallback

2026-07-22 12:55:21 +07:00

rustfmt.toml

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

scrape_media.py

feat: add Playwright browser scraping fallback for Instagram/Facebook/Twitter

2026-09-01 15:35:20 +07:00

stalk_tiktok.py

feat(stalk): add mobile-legends, free-fire, genshin, twitter, tiktok, instagram

2026-09-01 20:32:10 +07:00

README.md

Scraper API

Backend service berbasis Axum untuk scraping, image proxy/cache, dan endpoint API inti.

Stack

Rust + Axum
SeaORM (MySQL)
Redis (deadpool-redis)
Utoipa + Swagger UI

Quick Start

cargo run

Server bind ke 0.0.0.0:${PORT} dengan default port 4091.

Required Environment Variables

DATABASE_URL=mysql://asephs:hunterz@localhost:3306/sosmed
JWT_SECRET=change-me
REDIS_URL=redis://localhost:6379

Optional yang sering dipakai:

RUST_LOG=info
MINIO_ENDPOINT=
MINIO_BUCKET_NAME=
MINIO_ACCESS_KEY=
MINIO_SECRET_KEY=

Useful Endpoints

GET /docs - Swagger UI
GET /api-docs/openapi.json - OpenAPI JSON
GET /api/anime2/*
GET /api/komik/*
POST /api/proxy/image-cache
POST /api/proxy/image-cache/audit

Notes

Database akan dicek/dibuat saat startup jika DATABASE_URL bertipe MySQL.
Service ini juga menginisialisasi browser pool dan scheduler saat boot.

License

MIT