MythEclipse/scraper
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
52 Commits 1 Branch 0 Tags
278d4fe32a6fb56516425e07380beebb85ca8ea6
Commit Graph
7 Commits
Author SHA1 Message Date
asepharyana 278d4fe32a feat(tools): add whois(RDAP), iplocation, tinyurl, check-hosting, cek-resi, hargapangan 
Ported from Shirokami-API scraper/tool/*.js:
- /tool/whois - RDAP lookup (whois.com HTML .df-raw is dead/JS-rendered since 2026; RDAP returns structured register/expiry/registrar data)
- /tool/iplocation - ipapi.co JSON
- /tool/tinyurl - tinyurl.com API
- /tool/check-hosting - hosting-checker.net API
- /tool/cek-resi - cekresi.com AES-128-CBC encrypted form POST
- /tool/hargapangan - Bapanas government API (upstream unreachable from VPS)

Fixes: scraper crate has no :has-text() pseudo-class -> replaced history-table scan with 2-cell-row detection; Html held across .await makes future non-Send -> scoped doc parsing
 2026-09-01 20:12:52 +07:00
asepharyana 7ffc1bdd81 feat(search): add bmkg, jadwal-sholat, weather, yt-search, google endpoints 2026-09-01 19:51:34 +07:00
asepharyana 091a68bd55 feat(weebs): add anime/manga info, whatanime, waifu endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Ported from Shirokami-API scraper/weebs/*.js:
- /weebs/anime-info (jikan.moe anime search)
- /weebs/manga-info (jikan.moe manga search)
- /weebs/whatanime (trace.moe reverse image search)
- /weebs/sfw-waifu + /weebs/nsfw-waifu (waifu.pics)

Tested: anime-info (Naruto real data) + whatanime (real match). manga-info
+ waifu currently report upstream 504/DNS-down (jikan manga endpoint &
waifu.pics unreachable from VPS right now) - proxy logic verified via the
working anime/whatanime siblings.
 2026-09-01 19:35:32 +07:00
asepharyana 242e3498fe feat(stalk): add github + youtube profile stalk endpoints 
Ported from Shirokami-API scraper/stalk/{github,youtube}.js:
- /stalk/github (GitHub public API profile)
- /stalk/youtube (ytInitialData parse from /videos subpage:
  handles lockupViewModel (new), videoRenderer, gridVideoRenderer,
  shortsLockupViewModel; channel metadata + top-5 videos)

Also fix misc harga-emas main price to smallest-gram (1gr).
 2026-09-01 19:31:37 +07:00
asepharyana 6df8af54eb feat(misc): add misc endpoints (currency-converter, harga-emas, kurs-bca, server-info)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Ported from Shirokami-API scraper/misc/*.js:
- /misc/currency-converter (wise.com scrape, amount/from/to)
- /misc/harga-emas (harga-emas.org Antam table)
- /misc/kurs-bca (bca.co.id jual/beli)
- /misc/server-info (OS/CPU/RAM/disk from /proc)

Also: remove /download/tiktok/v2 + /download/twitter/v2 as separate routes;
douyin.wtf + twitsave now internal fallback steps in the primary
fetch_tiktok/fetch_twitter chains (v2 = fallback per user directive).
 2026-09-01 19:05:42 +07:00
asepharyana 763a641593 feat(downloader): add 26 downloader APIs + detect endpoint
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Port downloaders from Shirokami-API (Node.js):
- Instagram/Facebook (snapsave.app, downr.org fallback)
- TikTok (tikwm.com)
- YouTube MP4/MP3 (savetube.media, ydlp.yard.id)
- Spotify, Twitter/X (api.lrm.tube)
- Bilibili, Pinterest, MediaFire, Mega, TeraBox
- PixelDrain, Threads, DoodStream, KrakenFiles
- Danbooru, SoundCloud, Google Drive

Domain layer: DownloadResult + MediaItem entities
Application layer: 7 use-cases + detect_platform auto-detect
Infrastructure: DownloaderRepository (22 fetch functions)
Presentation: 21 handlers + DTO + OpenAPI schemas

Also: upgrade OpenTelemetry to 0.28 (axum 0.7 conflict resolution)
 2026-09-01 00:54:06 +07:00
asepharyanaandClaude Opus 4.8 776fa828d8 feat: full clean architecture refactor — domain → application → infrastructure → presentation 
- Restructure from Modular MVC to Clean Architecture with 4 strict layers
- Domain: entities (anime/komik/proxy), Repository traits, typed errors
- Application: use case classes with proper error propagation
- Infrastructure: repository impls, parsers, Redis cache, HTTP/scraping, browser
- Presentation: Axum handlers, DTOs, AppState, router, AppError+IntoResponse
- Migrate all parsers (otakudesu, alqanime, komik) to native infra implementations
- Replace once_cell::sync::Lazy/OnceCell with std::sync::LazyLock/OnceLock
- Remove 150+ old files in modules/ and shared/ directories
- Remove once_cell from Cargo.toml dependencies
- Fix test/debug binaries to use new import paths
- Zero new clippy warnings

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 12:54:23 +07:00