Pixiv support by sleroq · Pull Request #5 · divaltor/starlight

sleroq · 2026-08-15T13:44:49Z

This PR adds Pixiv as a first-class Starlight source: users can save individual artwork URLs and continuously sync their Pixiv bookmarks into the existing gallery and search experience.

Pixiv artwork URL / bookmark sync
  → Pixiv adapter
  → Post + Media
  → shared media collector
  → S3, duplicate detection, classification, embeddings
  → gallery and search

This PR also resolves important correctness problems in the existing Twitter pipeline

Bookmarks of the same image no longer leave stale media tasks. Previously, a duplicate image could be skipped after detection, leaving its new media record without the stored asset metadata needed to be considered complete. Now it is resolved against the existing asset: it reuses the S3 object, perceptual hash, and dimensions, then continues through classification.
Duplicate bookmarks now reuse an existing stored asset and metadata, keeping every user’s media record complete without redundant storage.
Incomplete media is collected again, and terminal classification failures are retried when encountered later.
Search and random discovery deduplicate repeated assets per user while preserving independently saved copies for other users.
Duplicate matching now evaluates candidates from capped hash-bucket queries instead of discarding full result sets.

Verification

46 tests passed.
Typecheck passed across all five packages.
Prisma migrations validated against an ephemeral PostgreSQL database.
Builds and Ultracite checks passed.

sleroq · 2026-08-15T14:27:14Z

-			unpublished: (chatId: number | string | bigint) => ({
-				deletedAt: null,
-				s3Path: { not: null },
-				publishedPhotos: { none: { chatId: Number(chatId) } },
-			}),
-		} satisfies Record<string, (...args: any) => PrismaGenerated.PhotoWhereInput>,


It was a stale method as far as I understand (codex told me)

sleroq · 2026-08-15T14:31:51Z

+				compute(data: { id: string; provider: string; userId: string }) {
+					if (data.provider !== "twitter") {
+						return `${data.provider}:${data.id}`;
+					}
 					// Split Twitter ID into 3 parts to handle large numbers that exceed bigint
 					const id = data.id;
 					const chunkSize = Math.ceil(id.length / 3);

 					const parts = [
 						id.slice(0, chunkSize),
 						id.slice(chunkSize, chunkSize * 2),
 						id.slice(chunkSize * 2),
 					].map((part) => Number.parseInt(part || "0", 10));

 					const userId = uuidParse(data.userId);

 					return sqids.encode([...parts, ...userId]);
 				},


Maybe we should move this logic outside of model methods as this stuff is source-specific.

But as you already had some logic there I decided not to complicate the PR further

sleroq · 2026-08-15T14:49:02Z

+  authorExternalId String? @map("author_external_id")
+  authorName       String? @map("author_name")
+  authorUsername   String? @map("author_username")
+  title            String?


I decided to expand the Post type, previously this stuff was stored inside tweetData JSON

sleroq · 2026-08-15T14:52:48Z

+  username            String? @unique
+  firstName           String  @map("first_name")
+  lastName            String? @map("last_name")
+  isBot               Boolean @default(false) @map("is_bot")


How could the user be a bot and why would we care but I digress.

sleroq · 2026-08-15T16:54:06Z

-		}
-
-		logger.info(
+		await classificationApp.spawn(


If this fails - we never retry and leave the media unclassified. This is unrelated to this PR, just letting you know.

sleroq · 2026-08-15T17:23:36Z

I think we should activate the source automatically after connection in the mini-app, instead of splitting this with scrapper command. But this is definitely outside of the scope of this PR.

sleroq · 2026-08-15T17:32:02Z

 	const isSubmitting =
 		saveCookiesMutation.isPending ||
 		deleteCookiesMutation.isPending ||
-		visibilityMutation.isPending;
+		visibilityMutation.isPending ||
+		savePixivMutation.isPending ||
+		deletePixivMutation.isPending ||
+		pixivPrivateMutation.isPending;
+	const twitterError = profile?.hasValidCookies
+		? deleteCookiesMutation.error?.message
+		: displayError;
+	const pixivError = profile?.hasPixivCredential
+		? (deletePixivMutation.error?.message ?? pixivPrivateMutation.error?.message)
+		: savePixivMutation.error?.message;


Fixed resurfacing of the twitter cookies save/delete errors as well btw

divaltor · 2026-08-15T17:35:43Z

nah

sleroq · 2026-08-15T19:49:47Z

+	for (const artwork of page.artworks) {
+		consecutiveKnown = known.has(artwork.id) ? consecutiveKnown + 1 : 0;
+		jobs.push({
+			userId: data.userId,
+			post: {
+				provider: "pixiv",
+				externalId: artwork.id,
+				sourceUrl: artwork.sourceUrl,
+				authorExternalId: artwork.author.id,
+				authorName: artwork.author.name,
+				authorUsername: artwork.author.username,
+				title: artwork.title,
+				text: artwork.caption,
+				tags: artwork.tags,
+				providerPayload: { starlightMediaType: artwork.type },
+				media: artwork.mediaUrls.map((url, position) => ({
+					externalId: `${artwork.id}:${position}`,
+					url,
+					position,
+					kind: artwork.type === "ugoira" ? "animation-preview" : "image",
+					fetchHeaders: { Referer: "https://www.pixiv.net/" },
+				})),
+			},
+		});
+		if (consecutiveKnown >= CONSECUTIVE_THRESHOLD) {
+			break;
+		}
+	}


Pixiv has an animation format called ugoria. I am not a fan.

So ugoira conversion only supported for direct Pixiv URLs, not from bookmarks. This way we have less code to support.

sleroq · 2026-08-15T21:09:31Z

Note that migration was edited by the agent, breaking the rules defined in the AGENTS.md

But it's for a good reason, Prisma cannot infer that:

tweets → posts
photos → media
tweet_data → provider_payload

Initial migration was generated with prisma, and only renames were manual.

Migration was tested on the main db in docker so I am pretty sure it's correct and good to go.

sleroq · 2026-08-15T21:12:52Z

Please consider merging this PR. It shouldn’t add much maintenance burden, as most of the changes simply abstract Twitter away as a provider rather than adding unnecessary new features that need ongoing support. I do use pixiv every week and if I ever stop, I can remove related code from the repo myself.

Also, please note that I spent a lot of time, effort, and Codex usage on this PR, trying to keep it to the highest standard and testing everything thoroughly. In case you still refuse to merge it, I will probably cry.

That said this PR also contain some fixes that needs to be ported separately, if this ends up not being merged.

sleroq · 2026-08-15T23:45:28Z

+import {
+	convertUgoira,
+	extractUgoiraZip,
+	MAX_PIXIV_DOWNLOAD_BYTES,
+	parsePixivArtworkUrl,
+} from "@/services/pixiv-media";
+import type { Context } from "@/types";
+
+const MAX_MANGA_BYTES = 150_000_000;


Currently we just throw if the post / media goes over the limit, but maybe we should pick downscaled version instead.

This would complicate the retry logic though

divaltor · 2026-08-16T12:37:23Z

+const createTemporaryDirectory = async () => {
+	const directory = `${Bun.env.TMPDIR ?? "/tmp"}/starlight-pixiv-${Bun.randomUUIDv7()}`;
+	const child = Bun.spawn(["mkdir", "-m", "700", directory], {
+		stdout: "ignore",
+		stderr: "ignore",
+	});
+	if ((await child.exited) !== 0) {
+		throw new Error("Failed to create Pixiv temporary directory");
+	}
+	return directory;
+};
+
+const removeTemporaryDirectory = async (directory: string) => {
+	const child = Bun.spawn(["rm", "-rf", directory], { stdout: "ignore", stderr: "ignore" });
+	await child.exited;
+};


divaltor · 2026-08-16T12:38:33Z

+		const user = ctx.user!;
+		const directory = await createTemporaryDirectory();
+		try {
+			const handled = await withPixivClient(user.id, async (client) => {


Why withPixivClient? What reason

divaltor · 2026-08-16T12:41:19Z

+		try {
+			const handled = await withPixivClient(user.id, async (client) => {
+				const artwork = await client.artwork(id);
+				if (artwork.type === "ugoira") {
+					const metadata = await client.ugoira(id);
+					const archive = await readResponseBounded(
+						await client.fetchMedia(metadata.zipUrls.medium),
+					);
+					const extracted = await extractUgoiraZip(archive, metadata.frames);
+					try {
+						const output = `${extracted.directory}/ugoira.mp4`;
+						await convertUgoira(extracted.concatPath, output);
+						await ctx.replyWithVideo(new InputFile(output), {
+							caption: artwork.title,
+						});
+					} finally {
+						await removeTemporaryDirectory(extracted.directory);
+					}
+					return true;
+				}
+
+				let aggregateBytes = 0;
+				const files: InputFile[] = [];
+				for (const [position, url] of artwork.mediaUrls.entries()) {
+					const bytes = await readResponseBounded(
+						await client.fetchMedia(url),
+						Math.min(MAX_PIXIV_DOWNLOAD_BYTES, MAX_MANGA_BYTES - aggregateBytes),
+					);
+					aggregateBytes += bytes.byteLength;
+					const extension = new URL(url).pathname.split(".").at(-1) ?? "jpg";
+					const path = `${directory}/${position}.${extension}`;
+					await Bun.write(path, bytes);
+					files.push(new InputFile(path));
+				}
+				for (let offset = 0; offset < files.length; offset += 10) {
+					const chunk = files.slice(offset, offset + 10);
+					const single = chunk.at(0);
+					if (chunk.length === 1 && single) {
+						await ctx.replyWithDocument(single, { caption: artwork.title });
+					} else {
+						await ctx.replyWithMediaGroup(
+							chunk.map((file, index) => ({
+								type: "document" as const,
+								media: file,
+								caption: index === 0 ? artwork.title : undefined,
+							})),
+						);
+					}
+				}
+				return true;
+			});
+			if (handled === undefined) {
+				await ctx.reply("Connect Pixiv in Settings first.");
+			}
+		} finally {
+			await removeTemporaryDirectory(directory);
+		}
+	},


Logic for Pixiv should be extracted into service, not be defined in Grammy handlers. As well as finding if there any ready-to-use libraries for Ugoira, using Bun.Archive or well-maintained libraries for archives (if any)

divaltor · 2026-08-16T12:43:04Z

+privateChat.command("scrapper").filter(
+	async (ctx) => {
+		const connections = await getScrapperConnections(ctx);
+		return connections.twitter === "connected" || connections.pixiv === "connected";
+	},
+	async (ctx) => {
+		const user = ctx.user!;
+		const connections = await getScrapperConnections(ctx);
+		const [twitter, pixiv] = await Promise.all([
+			connections.twitter === "connected"
+				? startTwitterCollection(user.id, ctx.update.update_id, ctx.logger)
+				: Promise.resolve(null),
+			connections.pixiv === "connected"
+				? startPixivCollection(user.id, ctx.update.update_id, ctx.logger)
+				: Promise.resolve(null),
+		]);
+		const results = [
+			twitter && { provider: "Twitter", ...twitter },
+			pixiv && { provider: "Pixiv", ...pixiv },
+		].filter((result) => result !== null);
+		const lookupFailures = [
+			connections.twitter === "lookup-failed" &&
+				"• Twitter: connection check failed; collection was not started.",
+			connections.pixiv === "lookup-failed" &&
+				"• Pixiv: connection check failed; collection was not started.",
+		].filter((line) => line !== false);
+		const complete =
+			lookupFailures.length === 0 &&
+			results.every((result) => result.immediateStarted && result.scheduleReady);
+		const lines = results.map(
+			(result) =>
+				`• ${result.provider}: sync ${result.immediateStarted ? "started" : "failed"}; recurring schedule ${result.scheduleReady ? "ready" : "failed"}.`,
+		);
+
+		await ctx.reply(
+			[
+				complete ? "Collection started:" : "Collection was only partially started:",
+				...lines,
+				...lookupFailures,
+				results.some((result) => result.immediateStarted)
+					? "Check your gallery in a few minutes."
+					: "No immediate sync started. Try /scrapper again.",
+			].join("\n"),
+			{ reply_markup: webAppKeyboard("app", "View gallery") },
+		);
+	},
+);


This could be extracted in separate repositroy\query with simple interface "ready" or something like that

divaltor · 2026-08-16T12:43:28Z

+interface ClassificationQueue {
+	spawn(
+		name: string,
+		data: { photoId: string; provider: string; userId: string },
+		options: {
+			idempotencyKey: string;
+			maxAttempts: number;
+			retryStrategy: unknown;
+		},
+	): Promise<{ created: boolean; taskID: string }>;
+	fetchTaskResult(taskID: string): Promise<{ state: string } | null | undefined>;
+	retryTask(taskID: string): Promise<unknown>;
+}
+
+interface ClassificationRecoveryDependencies {
+	classificationApp: ClassificationQueue;
+	retryStrategy: unknown;
+	logger: {
+		info(context: Record<string, unknown>, message: string): void;
+	};
+}
+
+export async function enqueueClassification(
+	{ classificationApp, retryStrategy, logger }: ClassificationRecoveryDependencies,
+	photoId: string,
+	provider: string,
+	userId: string,
+) {
+	const idempotencyKey = `classify-${provider}-${userId}-${photoId}`;
+	const task = await classificationApp.spawn(
+		"classification",
+		{ photoId, provider, userId },
+		{
+			idempotencyKey,
+			maxAttempts: 5,
+			retryStrategy,
+		},
+	);
+
+	if (task.created) {
+		return;
+	}
+
+	const result = await classificationApp.fetchTaskResult(task.taskID);
+	if (result?.state !== "failed") {
+		return;
+	}
+
+	// retryTask atomically changes the failed task back to pending. This keeps
+	// its idempotency key and lets every later terminal failure recover again.
+	// Omitting maxAttempts adds one attempt beyond the terminal task's count.
+	try {
+		await classificationApp.retryTask(task.taskID);
+	} catch (error) {
+		// Another collector may have recovered this task after our snapshot. Once
+		// it is no longer failed, that recovery is the coalesced outcome.
+		const latestResult = await classificationApp.fetchTaskResult(task.taskID);
+		if (!latestResult || latestResult.state === "failed") {
+			throw error;
+		}
+	}
+	logger.info(
+		{ photoId, provider, userId, taskId: task.taskID },
+		"Classification recovery enqueued",
+	);
+}


Retries out of scope of Pixiv

divaltor · 2026-08-16T12:49:38Z

+	const child = Bun.spawn(
+		[
+			"ffmpeg",
+			"-y",
+			"-f",
+			"concat",
+			"-safe",
+			"0",
+			"-i",
+			concatPath,
+			"-vf",
+			"scale=trunc(iw/2)*2:trunc(ih/2)*2",
+			"-c:v",
+			"libx264",
+			"-pix_fmt",
+			"yuv420p",
+			"-movflags",
+			"+faststart",
+			"-fs",
+			String(MAX_PIXIV_DOWNLOAD_BYTES),
+			output,
+		],
+		{ stdout: "ignore", stderr: "pipe" },
+	);


Why need ffmpeg to download an archive?

divaltor · 2026-08-16T12:49:49Z

+		const result = await Promise.race([
+			child.exited,
+			new Promise<"timeout">((resolve) => {
+				timeout = setTimeout(() => {
+					resolve("timeout");
+				}, FFMPEG_TIMEOUT_MS);
+			}),
+		]);
+		if (result === "timeout") {
+			child.kill();
+			await child.exited;
+			await stderr;
+			throw new Error("Ugoira conversion timed out");
+		}
+		await stderr;
+		if (result !== 0) {
+			throw new Error("Failed to convert ugoira");
+		}
+		if ((await Bun.file(output).stat()).size > MAX_PIXIV_DOWNLOAD_BYTES) {
+			throw new Error("Converted ugoira is too large");
+		}


We have EffectTS for that

divaltor · 2026-08-16T12:49:56Z

+import type { Tweet } from "@the-convocation/twitter-scraper";
+import { normalizeTags } from "@/services/tag-normalization";
+
+export const normalizeTwitterTags = (tweet: Pick<Tweet, "hashtags">): string[] =>


"normalize" are banned

divaltor · 2026-08-16T12:50:03Z

+export const normalizeTags = (tags: readonly string[]): string[] => {
+	const normalized: string[] = [];
+	const seen = new Set<string>();
+	for (const tag of tags) {
+		const value = tag.trim();
+		if (value && !seen.has(value)) {
+			seen.add(value);
+			normalized.push(value);
+		}
+	}
+	return normalized;
+};


"normalize" are banned

divaltor · 2026-08-16T12:50:50Z

+export const createPixivCredentialService =
+	<Client extends { refreshToken: string }>(dependencies: {
+		connect: (token: string) => Promise<Client>;
+		decryptLegacy: (secret: string, userId: string) => string;
+		decryptScoped: (secret: string, userId: string) => string;
+		encrypt: (token: string, userId: string) => string;
+		find: (userId: string) => Promise<{
+			credentialType: string;
+			encryptedSecret: string;
+		} | null>;
+		updateMatching: (
+			userId: string,
+			encryptedSecret: string,
+			replacement: string,
+		) => Promise<{ count: number }>;
+		withLock: <T>(userId: string, operation: () => Promise<T>) => Promise<T>;
+	}) =>
+	async <T>(userId: string, operation: (client: Client) => Promise<T>) => {
+		const client = await dependencies.withLock(userId, async () => {
+			const credential = await dependencies.find(userId);
+			if (!credential || credential.credentialType !== "refresh_token") {
+				return;
+			}
+
+			let token: string;
+			let migrated = false;
+			try {
+				token = dependencies.decryptScoped(credential.encryptedSecret, userId);
+			} catch {
+				token = dependencies.decryptLegacy(credential.encryptedSecret, userId);
+				migrated = true;
+			}
+
+			const client = await dependencies.connect(token);
+			if (migrated || client.refreshToken !== token) {
+				const updated = await dependencies.updateMatching(
+					userId,
+					credential.encryptedSecret,
+					dependencies.encrypt(client.refreshToken, userId),
+				);
+				if (updated.count === 0) {
+					throw new Error("Pixiv credential changed during token rotation");
+				}
+			}
+
+			return client;
+		});
+
+		if (!client) {
+			return;
+		}
+
+		return operation(client);
+	};


Why it's co complicated? Why we have decryptScope and decryptLegacy functions?

sleroq marked this pull request as draft August 15, 2026 13:46

sleroq force-pushed the feat/pixiv-support branch from 8c92e7e to 07e9dab Compare August 15, 2026 14:03

sleroq commented Aug 15, 2026

View reviewed changes

sleroq changed the title ~~feat: add Pixiv media and bookmark support~~ Pixiv support — plus a more reliable Starlight core Aug 15, 2026

sleroq changed the title ~~Pixiv support — plus a more reliable Starlight core~~ Pixiv support Aug 15, 2026

sleroq commented Aug 15, 2026

View reviewed changes

sleroq force-pushed the feat/pixiv-support branch from 1969d59 to f41bc9d Compare August 15, 2026 21:14

sleroq added 14 commits August 16, 2026 01:56

feat: add Pixiv media and bookmark support

c36ad9c

fix: tighten Pixiv credential and media handling

97f692a

fix: preserve Pixiv animation media kind

63a6d0c

refactor: rename post provider payload field

b9cf4cc

refactor: move Twitter cookies to provider credentials

c4afd6b

fix: harden provider credential migration

ebda045

fix: validate Twitter credential cookies

988b2c1

fix: support legacy Twitter cookie exports

ac126ca

fix: prevent root TypeScript emits

a472267

feat: make media search provider agnostic

54aa53e

Remove divider between sources in the UI

c09e8cf

fix: scope Pixiv crawl idempotency by run

ceb98a4

fix: preserve Pixiv schedule chain

d078cb5

fix: persist Pixiv token rotation before work

8eeccfc

sleroq added 16 commits August 16, 2026 01:58

fix: reject malformed public media ids

523d241

refactor: remove unused Pixiv force mode

ee1251d

fix: release Pixiv credential lock before work

7267a08

perf: parallelize Pixiv media queueing

5a8398d

fix: label Pixiv refresh token field

b07a3c6

fix: surface settings mutation errors

eb85d45

refactor: unify provider scraper activation

416e685

fix: order gallery media by position

8500043

fix: use app aliases in server tests

c19edff

refactor: adopt provider-neutral post naming

47abd3b

fix: resolve duplicate media per user

b565ead

refactor: consolidate media helpers

33f2511

fix: regenerate provider-neutral migration

860711e

refactor: use Bun APIs for ugoira files

a6cbf15

refactor: separate tag normalization concerns

23714db

fix(server): persist collected media without nested provider field

075e343

sleroq force-pushed the feat/pixiv-support branch from 0222454 to 075e343 Compare August 15, 2026 23:01

test: consolidate provider credential coverage

e11b176

sleroq commented Aug 15, 2026

View reviewed changes

fix: preserve pixiv search attribution

2efc2e7

divaltor requested changes Aug 16, 2026

View reviewed changes

refactor: simplify scraper command flow

bf24741

Conversation

sleroq commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR also resolves important correctness problems in the existing Twitter pipeline

Verification

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

divaltor commented Aug 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sleroq commented Aug 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sleroq commented Aug 15, 2026 •

edited

Loading