T

Deploy Scraper / build-and-deploy (push) Canceled after 0s

Details

fix: Twitter/X via Syndication API returns real MP4s (twitter-syndication provider)

Use cargo tokenless https://cdn.syndication.twimg.com/tweet-result?id={id}&token=0
which returns tweet JSON with video variants (MP4 at multiple bitrates).
Works server-side with no auth, wired as primary before savetwitter/yt-dlp.

2026-09-01 16:22:43 +07:00

.github

ci: deploy via Attic binary cache (attic.asepharyana.my.id) instead of ssh copy

2026-08-27 20:38:36 +07:00

.hermes/plans

fix: map ScrapingError to 502 Bad Gateway instead of 500 Internal Server Error; fix downr.org Cloudflare block; fix YouTube savetube DNS failure; fix twitter/pinterest error handling

2026-09-01 11:26:29 +07:00

docs/superpowers

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

scripts

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

src

fix: Twitter/X via Syndication API returns real MP4s (twitter-syndication provider)

2026-09-01 16:22:43 +07:00

tests

refactor: migrate thread/async/queue/scheduler infra to mytheclipse

2026-08-30 19:40:01 +07:00

tools/auto-lint

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

.gitignore

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

AGENT.md

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

Cargo.lock

feat(downloader): add 26 downloader APIs + detect endpoint

2026-09-01 00:54:06 +07:00

Cargo.toml

feat(downloader): add 26 downloader APIs + detect endpoint

2026-09-01 00:54:06 +07:00

check_image_cache_db.sh

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

CLAUDE.md

docs: update CLAUDE.md alqanime.si -> alqanime.net

2026-07-22 13:43:11 +07:00

ecosystem.config.cjs

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

flake.lock

fix: replace dead YouTube APIs with yt-dlp subprocess; fix all JSON panic crashes; add yt-dlp helper functions

2026-09-01 12:51:52 +07:00

flake.nix

fix: make scrape_media.py path resolution robust + include it in Nix build

2026-09-01 15:40:57 +07:00

GEMINI.md

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

package.json

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

README.md

refactor: remove EXTERNAL_BROWSERLESS_WS / browserless fallback

2026-07-22 12:55:21 +07:00

rustfmt.toml

chore: initial commit for asepharyana-hub-scraper

2026-07-09 22:07:03 +07:00

scrape_media.py

feat: add Playwright browser scraping fallback for Instagram/Facebook/Twitter

2026-09-01 15:35:20 +07:00

README.md

Scraper API

Backend service berbasis Axum untuk scraping, image proxy/cache, dan endpoint API inti.

Stack

Rust + Axum
SeaORM (MySQL)
Redis (deadpool-redis)
Utoipa + Swagger UI

Quick Start

cargo run

Server bind ke 0.0.0.0:${PORT} dengan default port 4091.

Required Environment Variables

DATABASE_URL=mysql://asephs:hunterz@localhost:3306/sosmed
JWT_SECRET=change-me
REDIS_URL=redis://localhost:6379

Optional yang sering dipakai:

RUST_LOG=info
MINIO_ENDPOINT=
MINIO_BUCKET_NAME=
MINIO_ACCESS_KEY=
MINIO_SECRET_KEY=

Useful Endpoints

GET /docs - Swagger UI
GET /api-docs/openapi.json - OpenAPI JSON
GET /api/anime2/*
GET /api/komik/*
POST /api/proxy/image-cache
POST /api/proxy/image-cache/audit

Notes

Database akan dicek/dibuat saat startup jika DATABASE_URL bertipe MySQL.
Service ini juga menginisialisasi browser pool dan scheduler saat boot.

License

MIT