feat: anime-sdk 2.0 — clean SDK surface, 9 verbs, opaque IDs, progressive search#9
Open
hexxt-git wants to merge 19 commits into
Open
feat: anime-sdk 2.0 — clean SDK surface, 9 verbs, opaque IDs, progressive search#9hexxt-git wants to merge 19 commits into
hexxt-git wants to merge 19 commits into
Conversation
…r wiring - Add Browse and Media pages to the example website; route / now lands on Browse - Wire AnilistMeta, MalMeta, KitsuMeta (shared MappingClient) into server.mjs - Add examples/cli — React Ink TUI with browse, search, media info, and stream resolution - Extend api.ts with meta routes (metaSearch, metaInfo, metaContent, metaStream, metaBrowse) and matching TypeScript types - Add shared UI components (Button, Collapsible, Input, Select) - Update breadcrumb logic in Layout to cover the new meta-aware routing - Fix dom.test.ts: querySelector searches children, so wrap target in a parent element - Add RESTRUCTURE.md design-proposal document
Move transport/ plumbing into src/internal/ (http, dom, hls, rateLimiter, retry, transport), urn helpers to src/internal/id.ts, and MappingClient to src/internal/mapping.ts. Old locations replaced with re-export stubs so existing tests and imports continue to work unchanged. Add empty scaffold files: sdk.ts, types.ts, errors.ts, config.ts, registry.ts, progressive.ts, health.ts — placeholders for Phases 2–7. Full unit test suite (94 tests) passes. tsc --noEmit clean.
linkedom is already in dependencies. dom.ts already auto-registers it via the top-level globalThis.DOMParser assignment on module import, guarded by a typeof check so it skips browser/jsdom environments. No E2E setup shim existed to delete. The existing dom.test.ts unit test verifies that BrowserDomParser works in Node without any manual setup. Phase 1 done: consumers never need to touch globalThis.DOMParser.
Add src/types.ts: Media, Episode, Chapter, Stream, Pages, List<T>, SourceInfo, MediaTitle, MediaCover, Subtitle, Score — all plain POJOs. Add src/errors.ts: AniError extends Error with AniErrorCode const enum. Add src/config.ts: SdkOptions type + resolveOptions() with defaults. Add tests/types.test.ts: JSON round-trips, error code branching, default option merging (9 tests, all pass). tsc --noEmit clean.
Add encodeId/decodeId to src/internal/id.ts. Plain base64url-encoded JSON
tokens: { v:1, t:'media'|'episode'|'chapter', s:sourceId, r:rawId, m? }.
decodeId throws AniError(BadId) on malformed input, missing required fields,
or wrong version.
Legacy URN helpers kept in the same file for existing providers (removed in
Phase 9). tests/id.test.ts: 7 tests covering round-trip, mappings,
malformed input, version check. tsc clean. 110 unit tests pass.
… AniList source Add src/sources/base.ts: single Source interface (internal) with caps flags and optional capability methods — replaces BaseProvider+BaseMetadataProvider for new sources. Add src/health.ts: HealthTracker — rolling 20-call success/latency window per source. Synchronous snapshot(). Used to rank playback sources. Add src/registry.ts: Registry.register(), sourcesFor(), fanOutSearch() (AsyncIterable), mergeEpisodes(), rankPlaybackSources(). MappingClient invocations wired in Phase 6 when playback sources migrate. Add src/sources/anilist.ts: AnilistSource implements Source. Ports AniList GraphQL search/info/browse from AnilistMeta with new Media return type and opaque base64url IDs. Unit tests (5): registry fanout, sourcesFor filtering, health stats. Live E2E (4): search/info/browse all pass against graphql.anilist.co.
Add src/progressive.ts: createProgressiveResult<T>() returns an object that is both AsyncIterable<T> (results as they arrive) and PromiseLike<T[]> (collect all). AbortSignal.any() composes caller signal with internal cancel(). Aborted result rejects with AniError(Cancelled). Update src/registry.ts: fanOutSearch() now returns ProgressiveResult<Media> via createProgressiveResult — each source is a producer that forwards its AbortSignal. AbortSignal is passed through every fanOutSearch call. tests/progressive.test.ts: 5 tests — await collect, async iteration, cancel via AbortSignal, cancel() stops iteration, empty producers. 120 unit tests pass. tsc clean.
Add src/sources/ for all remaining content+catalogue providers: - Catalogue: mal.ts (Jikan), kitsu.ts (JSON:API) — search/info/browse - Playback anime: allmanga.ts, megaplay.ts, animeparadise.ts, anikoto.ts, gogoanime.ts, goyabu.ts — episodes/stream - Playback manga: mangadex.ts, weebcentral.ts, mangapill.ts — chapters/pages All sources: - Implement Source interface (src/sources/base.ts) - Use encodeId/decodeId for opaque base64url IDs - Return Media/Episode/Chapter/Stream/Pages (new 2.0 types) - Pass AbortSignal to every HTTP call Add streamToPayload() helper in screenshotHelper.ts for E2E compat. Update E2E tests for all 11 sources to use new *Source classes. Delete obsolete tests: anilistMeta, baseMetadataProvider, mappingClient, providerUrnRoundTrip (replaced by new source tests or obsolete). Old providers/meta files kept as re-export stubs — deleted in Phase 10. tsc clean, 120 unit tests pass.
Add src/sdk.ts: Sdk class with 9 verbs (search, info, sources, episodes, chapters, stream, pages, browse, health). createSdk(opts?) instantiates an HttpClient, builds enabled sources, registers them in the Registry. Each verb dispatches to the right source via decodeId(). Search returns ProgressiveResult from the registry's fanOutSearch. Update src/index.ts: export createSdk, Sdk, Media, Episode, Chapter, Stream, Pages, List, SourceInfo, Score, AniError, AniErrorCode, SdkOptions as the new 2.0 public surface alongside legacy 1.x exports (removed in Phase 10). tests/sdk.test.ts: 5 unit tests — zero-config init, health snapshot, source filtering, ProgressiveResult shape, AniError export. All pass. tsc clean.
Add src/server/routes.ts: 9 routes mirroring SDK verbs (search, media,
episodes, chapters, sources, episode stream, chapter pages, browse, health).
Each handler is ~10 lines: parse params → call SDK → JSON-serialize.
AbortController bridges req.close() to SDK cancellation.
Add src/server/cli.ts: env-driven process entry — reads PORT,
SOURCES_DISABLED, --help flag, starts server via startServerV2().
Add startServerV2({ port, sdk }) to src/server/index.ts: creates SDK
(or uses provided), builds routes, listens. Old startServer() kept intact
for existing tests.
Add bin entry: "anime-sdk": "./dist/server/cli.js"
Add ./server export in package.json for startServerV2 import path.
tests/e2e/server_v2.test.ts: 5 integration tests — health, search,
browse, 400 on missing q, 404 on unknown route. tsc clean, 125 tests pass.
examples/server.mjs: one-liner — createSdk() + startServerV2() replaces
50-line provider/meta wiring. Zero config, all sources enabled by default.
examples/website/src/api.ts: use new server routes (/search, /media/:id,
/episode/:id/stream, etc.). Define Media/Episode/Chapter/Stream/Pages types
matching the SDK. Drop CONTENT_PROVIDERS/META_PROVIDERS constants, old
meta/content split, and duplicated type definitions. Add formatScore() that
consumes Score{value,scale} without dividing by 10 everywhere.
examples/website/src/pages/Search.tsx: unified search (no Catalogue/Provider
toggle). Kind selector (anime/manga) only. Kills bug #1.
examples/website/src/pages/Browse.tsx: uses new browse() with List<Media>.
Drop meta provider picker. Kind selector only.
examples/website/src/pages/Media.tsx: uses mediaInfo() + mediaSources().
Shows available sources from SDK (kills bug #4). Score via formatScore().
Drop character/staff/relation sections (not in Media type).
examples/website/src/pages/Episodes.tsx: single code path for episodes
and chapters. Episode id is opaque — no display-label parsing (kills bug #3).
Unified route /stream?epid= or /stream?chid=.
examples/website/src/pages/Stream.tsx: uses episodeStream()/chapterPages().
Adjacent prev/next from stream.adjacent (kills bug #8 refetch).
origin.host from stream.origin.host (kills bug #7 proxy URL parsing).
Manga has no language argument (kills bug #5).
Update src/index.ts: 2.0 surface is now the primary export (createSdk, Sdk, types, AniError, AniErrorCode, SdkOptions, startServerV2). Legacy 1.x exports moved to a clearly-marked backward-compat section — old providers, meta classes, transport classes, and utilities kept temporarily for the proxy/download tests; these will be removed when those tests are updated to the 2.0 API. Delete tests/e2e/concurrencyCap.test.ts (tests BaseProvider.maxConcurrency which is a 1.x-only concept). Delete tests/e2e/metaIntegration.test.ts (tests the old BaseMetadataProvider + MappingClient + BaseProvider chain; replaced by new source integration tests). tsc clean, 125 unit tests pass.
README.md: rewritten against 2.0 API with three code samples (search→stream, server one-liner, custom config), error handling, cancellation, source table, server routes table. CLAUDE.md: updated to reflect new architecture — single Source interface, internal/id.ts for opaque IDs, MappingClient is private, no DOMParser shim, Registry+Sdk+ProgressiveResult layer. Updated source-addition guide. .changeset/sdk-2-0.md: major bump with migration table (old → new API). Build passes (272 KB ESM). 125 unit tests pass. tsc clean.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The registry's mergeEpisodes() was only checking media.mappings.sources?.[src.id] which is never populated when media comes from a catalogue source (AniList, MAL, Kitsu) — it has no AllManga/MegaPlay source ID baked in. Add resolveMediaId() to Registry: checks cached mappings.sources first, then calls source.lookupByMapping() if the source has the mapping capability. Caches the result in media.mappings.sources for subsequent calls. Add mergeChapters() to Registry (parallel to mergeEpisodes) and use it in Sdk. Fix AllmangaSource.lookupByMapping: was incorrectly searching by AniList numeric ID as text — AllManga has no native AniList lookup, return null. Fix MangadexSource.lookupByMapping: was using ids[] with a MAL ID — MangaDex ids[] expects UUIDs, not MAL IDs, return null. MegaPlaySource.lookupByMapping correctly returns the AniList ID (MegaPlay indexes by AniList ID natively) — this one works. sdk.episodes(anilistMedia) now works for MegaPlay; AllManga cross-source resolution still requires a title-based MALSync lookup (out of scope for Phase 6 — existing MappingClient in src/internal/mapping.ts handles this for the legacy server but isn't wired to the new Source interface yet).
Marketing website (Astro):
- Demo.astro: createSdk() + sdk.search/episodes/stream instead of MegaPlayProvider
- Server.astro: npx anime-sdk + startServerV2() instead of startServer({ providers })
- Proxy.astro: createSdk({ proxy }) instead of startServer({ providers, proxy })
- Contribute.astro: Source interface + createSdk() instead of BaseProvider
- CodeEditorScene.tsx: createSdk() typing animation instead of AllmangaProvider
- TerminalScene.tsx: sdk.episodes/stream instead of fetchContentUnits/resolveStream
- ProviderDashboardScene.tsx: 'allmanga' + stream.url instead of AllmangaProvider/resolveStream
- faq.astro: sdk.stream(), sources/disabled config instead of provider-specific guidance
- og/[...slug].png.ts: updated type names and source count (12 not 9)
examples/cli.mjs: rewrite to createSdk(); covers anime+manga, uses
sdk.search/episodes/chapters/stream/pages; prints stream.url, origin.host, adjacent.
examples/cli/index.tsx: full Ink TUI rewrite (1147→370 lines). Uses createSdk()
directly — no HTTP, no separate meta/content providers, no provider selection
screen. Screens: home → browse/search → results → media → episodes/chapters →
stream/pages. Episode ids are opaque (no display-label parsing).
All references to 1.x API (BaseProvider, fetchContentUnits, resolveStream,
AllmangaProvider, HttpClient exports, IVideoPayload, startServer with providers
array, URN strings, 9 providers) removed or updated to 2.0 equivalents.
Getting Started (index.mdx): createSdk() → search/episodes/stream flow,
plain POJO types, progressive search, AniError, download utilities.
HTTP Server (http-server.mdx): startServerV2 / npx anime-sdk, all 9 routes
(/search, /media/:id, /episodes, /chapters, /sources, /episode/:id/stream,
/chapter/:id/pages, /browse, /health) with request/response shapes.
API Reference (api-reference.mdx): complete 2.0 surface — createSdk,
SdkOptions, Sdk methods, all value types (Media, Episode, Chapter, Stream,
Pages, List, SourceInfo, Score), AniError/AniErrorCode, startServerV2,
downloadVideo/downloadMangaChapter.
Contributing (contributing.mdx): Source interface instead of BaseProvider,
encodeId/decodeId, streamToPayload, updated checklist.
Proxy (proxy.mdx): createSdk({ proxy }) instead of startServer + proxy: true,
stream.url / stream.origin.host / stream.origin.proxied.
Download (download.mdx): sdk.stream() → downloadVideo, sdk.pages() →
downloadMangaChapter.
Providers overview: renamed "Providers" → "Sources", updated tables, 12 total.
All 12 individual source docs: createSdk({ sources: [...] }) pattern, removed
old Provider constructor usage.
Hero: "9 providers" → "12 sources", updated description.
HeroStats: "9 Providers" → "12 Sources".
Layout default description: updated.
- Remove src/providers, src/meta, src/transport, src/utils/urn, src/internal/mapping, src/download (legacy 1.x surface). - Collapse src/server/index.ts to the v2 server; rename startServerV2 → startServer, ServerV2Options → ServerOptions. - Trim src/types/index.ts and src/index.ts to only what 2.0 needs. - Update tests, README, CLAUDE.md, examples, and the website docs to drop references to removed APIs.
…split Core SDK changes: - Add `src/download/` module — downloadVideo (HLS→MP4 via ffmpeg), downloadMangaPage (single image), downloadMangaChapter (pages→ZIP); exported from src/index.ts - Add `src/internal/similarity.ts` — token-based fuzzy title similarity used by the registry - Add `src/server/proxy.ts` — SSRF-safe HTTP proxy with optional HMAC signing, host allowlist, and HLS manifest rewriting; exposed on /proxy when ServerOptions.proxy is set - Add `HttpClient.withHeaders()` — clone an HttpClient sharing the same rate-limiter/transport but with extra headers, used to give browser-UA sources their own client without forking the rate budget - Registry.resolveMediaId: add fuzzy title-search fallback (step 3) — when lookupByMapping returns null and the source has `caps.search`, search by title and pick the best candidate above a 0.7 similarity threshold; year guard prevents "Naruto" matching "Naruto: Shippuden" - Registry.rankPlaybackSources: run source probes in parallel (was sequential) - sdk.ts: centralize browser User-Agent in buildSources(); browser-heavy sources (allmanga, megaplay, anikoto, gogoanime, goyabu, weebcentral) get browserHttp; API sources (anilist, mal, kitsu, mangadex, mangapill) keep plain http - Sources (anikoto, gogoanime, goyabu, megaplay): remove per-constructor UA mutation now that sdk.ts owns the split - Sources (kitsu, mal): info() now receives decoded `r` field directly (Sdk.info decodes first); drop redundant decodeId call in the method body - Server: add CORS headers to all JSON responses and OPTIONS preflight; add /proxy route; add SSE-based download endpoints (/download/video/progress+file, /download/manga/chapter/progress+file) with 10-minute temp file cleanup - cli.ts: parse PROXY / PROXY_BASE / PROXY_SIGN_SECRET / PROXY_ALLOWED_HOSTS env vars and thread through to startServer - Tests: add registry fuzzy-resolution tests, download unit tests, proxy unit/E2E tests, sdkInfo and sdkSources E2E suites, similarity unit tests - Docs/website: add proxy and download pages, update API reference, add Proxy section to marketing site
- Refactor `Source.stream` interface to return a list of `Stream` candidates (`Stream[]`) rather than a single nested object. - Update all anime playback sources (allmanga, anikoto, animeparadise, gogoanime, goyabu, megaplay) to return arrays of stream candidates. - Implement progressive fanning-out of stream resolution across sources in `Registry` and `Sdk`, returning a `ProgressiveResult<Stream>`. - Add `/episode/:id/streams` Server-Sent Events (SSE) server endpoint for progressive streaming of candidates to the client. - Rewrite example web application page (Stream.tsx) and API layer to consume the new EventSource stream endpoint and present candidates grouped by language, server, and quality. - Update CLI, TUI, documentation (README, CLAUDE.md, website mdx), and test suites to reflect the progressive stream API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full rewrite of anime-sdk following the plan in PLAN.md and design in RESTRUCTURE.md. 12 phases, each ending green.
What changed
createSdk(): zero-config factory. All 12 sources, built-in DOM parser.search,info,sources,episodes,chapters,stream,pages,browse,health.Media,Episode,Chapter,Stream,Pages,List<T>—JSON.stringify-safe, opaqueidfields.ProgressiveResult<T>: search returnsAsyncIterable<T> & PromiseLike<T[]>— iterate as results arrive orawaitthe full array.AniError+AniErrorCode: structured errors, branch on.codenot.message.Stream.adjacent: prev/next without a second fetch (kills bug n8).Stream.origin.host: no proxy URL parsing (kills bug n7).{ value, scale }: no more(score / 10).toFixed(1)(kills bug n6).sdk.pages(chapter)— no language arg (kills bug n5).sdk.sources(media): ranked playable providers — safe "Watch via" dropdown (kills bug n4).npx anime-sdk: zero-install server.Sourceinterface: replacesBaseProvider+BaseMetadataProvider.src/internal/: all plumbing private —HttpClient,MappingClient, extractors, HLS utils.src/index.ts; removed in 3.0.Phases
feat(scaffold)feat(dom)feat(types)feat(ids)feat(registry)feat(progressive)feat(sources)feat(sdk)feat(server)feat(examples)feat(cleanup)docs(2.0)