diff --git a/templates/clips/.agents/skills/video-sharing/SKILL.md b/templates/clips/.agents/skills/video-sharing/SKILL.md index 0851f2ada1f..e4b5c0eee86 100644 --- a/templates/clips/.agents/skills/video-sharing/SKILL.md +++ b/templates/clips/.agents/skills/video-sharing/SKILL.md @@ -143,6 +143,11 @@ A Clip placed in a Content document has one `recording_content_bindings` row per - The token is signed with `CONTENT_CLIPS_PEER_SECRET` (`clips.contentPeerSecret`, read with `resolveContentClipsPeerSecret`), a secret only the Content and Clips deployments hold, and Clips verifies it with that secret alone. That is what makes the token's issuer trustworthy as the Content app: a token signed with the deployment `A2A_SECRET` or a workspace secret is refused, because other apps hold those. While the secret is unset, shorter than 32 characters, or equal to `A2A_SECRET`, every route answers 503 with code `content_clips_not_configured`; set it on both deployments. - Playback returns Clips media URLs with a five-minute `content_grant` that names the reader, not a role. The grant is signed with a key derived from `CONTENT_CLIPS_PEER_SECRET`, never the app's own server secret (which a workspace derives from `A2A_SECRET`), and a media request carrying a grant while the secret is unusable gets the same 503. `/api/video` and `/api/thumbnail` re-run every gate on each request, so trashing the binding stops a URL already handed out. An expired grant is a 403 with code `content_grant_expired`; Content's player then asks Content for playback again, which re-checks document access. - Each reader's reads and writes of one Clip through one document are rate-limited per process. A request counts against the Clip's bucket only once Clips has opened it: Content mints a token for any recording id a page reader names, so counting earlier would let a reader name made-up ids until their own spent bucket dropped out. A full map drops its oldest bucket rather than refuse a new one. A request whose Clip doesn't open counts as a miss against that reader's misses for that Clip and document; once they are spent, the Clip is refused before any database read. Misses are kept per Clip so made-up ids can't use up a real Clip's, which leaves a caller who keeps naming new made-up ids unbounded here: each costs one lookup, and Content's own per-visitor limit has to bound it. Every signed-out reader of a document shares the public reader's buckets. +- A Content editor's upload uses the `content-clip:upload` scope, whose token carries `document_id`, the editor's `name`, and the browser's `client_ip` when Content knows it, never a `recording_id`. Content checks that the reader can edit the page before minting it. `GET /api/content-clips/upload-status` answers the editor's `storage` state (`ready`, `free` with what's left, `used_up`, `none`, or `account_required` when the email has no Clips account or workspace; Clips never makes either). `POST /api/content-clips/uploads` adds the `upload` binding first, then a private recording the editor owns, and returns a ticketed `chunkUrl` and `cancelUrl`; a state that can't take the file is a 409 `storage_required` with that `storage`. The start carries the `uploadKey` Content makes once per picked file, kept on the binding with the file's type and size: a start repeated with the same key, page, and editor never makes a second Clip. It answers that upload with another ticket and the first start's plan while it is uploading, a 409 `upload_already_finished` with `recordingId` and `status` once a final chunk holds it or Clips is making the Clip, a 409 `upload_starting` while the latest start is still making its recording (Content repeats it after a few seconds), or a 409 `upload_key_used` when it was cancelled or failed, is parked until storage is connected, began in another block, or describes a file of another type or size. A start that fails deletes its binding only when nothing can be left: a refusal raised before any recording or free clip (`refusedStart`), or an error with `cleanupConfirmed: true`. Any other failure keeps the binding and marks `upload_start_failed_at`, and a repeat that finds no recording under a marked binding, or one whose latest attempt is `RESERVATION_GRACE_MS` old, takes the start over with a compare-and-swap on the binding (`takeOverStart`) and makes the recording under the binding's own recording id, so the free clip reserved per recording id is never held twice; of repeats that overlap, the rest get `upload_starting`. A kept binding whose start left a recording answers from where that upload stands, so one whose recording failed answers `upload_key_used` like a cancelled one; that key's upload ended, and Content starts the file again under a new key. Earlier tickets keep working, since overlapping repeats each hand one back: a recording keeps its four newest (`addRecordingUploadTicket`) and still holds one upload slot, while an agent's `create-recording-upload` keeps one. A repeat skips the storage check, since its first start may have taken the last free clip. +- The editor's browser posts the file to `/api/uploads/:recordingId/content-chunk` with the ticket in the query, no cookie, and no custom headers, so the request is a CORS simple request and no preflight reaches the auth guard. Only the paired built-in Content's origin (`pairedContentOrigin`) gets `Access-Control-Allow-Origin`, never the request's own `Origin`, and the answer never allows credentials. +- When the page gives an upload up (its block is gone, the editor can no longer edit, or a chunk failed for good), the browser POSTs `cancelUrl` (`/api/uploads/:recordingId/content-cancel?ticket=…`) the same way. Clips' own abort fails the recording and releases its free clip, as the lease reaper would an hour later; the `upload` binding stays. Where an upload stands is one answer, `readUploadPhase` in `server/lib/final-chunk-claim.ts`: `uploading`, `finishing` (a final chunk holds it), `parked` (Clips has the whole file and waits for storage), `finished` (`processing` or `ready`) or `ended` (cancelled or failed). The repeated start, the cancel and the Content chunk route all use it, and nothing else reads its three stores (the final chunk's claim, the upload state and the recording row) together. It reads the claim first, then the upload state, then the row. Before Clips takes the final chunk's bytes, on the buffered and the resumable path alike, the Content chunk route renews the row's lease to outlast its claim, only while the row is `uploading`, then claims the upload, changing its upload state with the same compare-and-swap the abort claims with; once Clips accepts the chunk it writes `parked` or `finished` into the upload state, and only then gives the claim up, so a reader that finds no claim finds where the upload went. A final chunk that fails gives the claim up with the upload still `uploading`. A claim older than `FINAL_CHUNK_CLAIM_TTL_MS` (a little over Clips' 75 s function timeout) was left by a request that was cut off, so the upload reads `uploading` again and the next final chunk or cancel takes it over. A live claim is never taken over: a cancel or another send of the final chunk changes nothing while the upload is `finishing` and answers 409 `upload_finishing` (`CONTENT_CLIP_CHUNK_FINISHING` for the chunk), which Content repeats every few seconds, the chunk for a little longer than the claim lasts. A final chunk sent once the upload is `parked` or `finished` answers the first one's success (202 `waitingForStorage`, or 200 with `status`) and Clips takes no bytes; one that finds the upload `ended` is refused (409). The answer's `outcome` comes from the phase read after the abort, never from the abort's `ok`: `cancelled` (200) once it is `ended`, a repeat included and a parked upload too, or `already_finished` (409, with `status`) once it is `finished`. Any other answer means it wasn't cancelled; an upload whose cancels keep finding `finishing` is left to the reaper. The reaper reads only the row's lease, never the phase: it fails a row whose lease ran out in one conditional update, so it either failed the row before the final chunk renewed it, and the chunk finds it `ended`, or it leaves the row. A page in Trash can still cancel. A closed tab is still left to the reaper. +- Upload-origin Clips are listed under From Content (`list-recordings` view `from-content`, each with `contentPageUrl` back to its page), and Library's own list leaves them out. A folder someone moved one into still lists it, and linked Clips stay in Library. +- `isActiveUploadBinding` / `activeUploadBinding()` in `server/lib/content-uploads.ts` is the one test for an active upload binding. Playback's grant, the chunk route, From Content and its page links all use it, so an upload whose page is in Trash plays for no document reader, takes no chunks (409), and lists in neither From Content nor Library. - Create and change binding rows only through Content's upload and lifecycle flows. ## Embed URLs diff --git a/templates/clips/actions/create-recording-upload.test.ts b/templates/clips/actions/create-recording-upload.test.ts index 0b1c7d2206a..55c7fee10c2 100644 --- a/templates/clips/actions/create-recording-upload.test.ts +++ b/templates/clips/actions/create-recording-upload.test.ts @@ -158,6 +158,7 @@ describe("create-recording-upload", () => { expect(mocks.releaseRecordingUploadTicket).toHaveBeenCalledWith( "owner@example.com", "rec-1", + "hash", ); }); diff --git a/templates/clips/actions/create-recording-upload.ts b/templates/clips/actions/create-recording-upload.ts index d87c0966dfd..f3da0fa6d8c 100644 --- a/templates/clips/actions/create-recording-upload.ts +++ b/templates/clips/actions/create-recording-upload.ts @@ -3,17 +3,11 @@ import { getAppProductionUrl, getRequestContext, } from "@agent-native/core/server"; -import { UPLOAD_SLICE_BYTES } from "@shared/recording-core.js"; import { formatUploadBytes, MAX_UPLOAD_BYTES } from "@shared/upload-limits.js"; import { z } from "zod"; import { getServerAppBasePath } from "../server/lib/public-agent-context.js"; -import { - createRecordingUploadTicket, - releaseRecordingUploadTicket, -} from "../server/lib/recording-upload-ticket.js"; -import { getCurrentOwnerEmail, nanoid } from "../server/lib/recordings.js"; -import createRecording from "./create-recording.js"; +import { startRecordingUpload } from "../server/lib/recording-upload-start.js"; const MAX_UPLOAD_SIZE = formatUploadBytes(MAX_UPLOAD_BYTES, "down"); @@ -78,78 +72,35 @@ export default defineAction({ .describe("Video height in pixels."), }), run: async (args, actionContext) => { - const ownerEmail = getCurrentOwnerEmail(); - // The ticket comes first: if it is refused or fails to save, no recording - // is left stranded in "uploading" with nothing able to finish it. - const recordingId = nanoid(); - const ticket = await createRecordingUploadTicket( - ownerEmail, - recordingId, - getRequestContext()?.authUserId, + const upload = await startRecordingUpload( + { + title: args.title, + visibility: args.visibility, + mimeType: args.mimeType, + sizeBytes: args.sizeBytes, + durationMs: args.durationMs, + width: args.width, + height: args.height, + recordingSource: "agent_upload", + }, + actionContext, ); - let created: Awaited>; - try { - created = await createRecording.run( - { - id: recordingId, - title: args.title, - visibility: args.visibility, - mimeType: args.mimeType, - sizeBytes: args.sizeBytes, - width: args.width, - height: args.height, - hasAudio: true, - hasCamera: false, - recordingSource: "agent_upload", - requestStreaming: true, - }, - actionContext, - ); - } catch (error) { - // create-recording fails its own half-made row, or the lease reaper does; - // only the ticket, which the agent never received, is ours to release. - await releaseRecordingUploadTicket(ownerEmail, recordingId).catch( - (releaseError) => { - console.warn("[create-recording-upload] ticket release failed", { - recordingId, - error: - releaseError instanceof Error - ? releaseError.message - : String(releaseError), - }); - }, - ); - throw error; - } - + const { recordingId, totalChunks } = upload; const appUrl = `${new URL(getRequestContext()?.requestOrigin || getAppProductionUrl()).origin}${getServerAppBasePath()}`; - const totalChunks = Math.max( - 1, - Math.ceil(args.sizeBytes / UPLOAD_SLICE_BYTES), - ); - const query = new URLSearchParams({ - mimeType: args.mimeType, - total: String(totalChunks), - hasAudio: "1", - hasCamera: "0", - }); - if (args.durationMs) query.set("durationMs", String(args.durationMs)); - if (args.width) query.set("width", String(args.width)); - if (args.height) query.set("height", String(args.height)); return { - recordingId: created.id, + recordingId, visibility: args.visibility, - shareUrl: `${appUrl}/share/${encodeURIComponent(created.id)}`, + shareUrl: `${appUrl}/share/${encodeURIComponent(recordingId)}`, method: "POST" as const, - chunkUrl: `${appUrl}/api/uploads/${encodeURIComponent(created.id)}/agent-chunk?${query.toString()}`, + chunkUrl: `${appUrl}/api/uploads/${encodeURIComponent(recordingId)}/agent-chunk?${upload.chunkQuery.toString()}`, headers: { - Authorization: `Bearer ${ticket.token}`, + Authorization: `Bearer ${upload.ticket}`, "Content-Type": args.mimeType, }, - chunkSizeBytes: UPLOAD_SLICE_BYTES, + chunkSizeBytes: upload.chunkSizeBytes, totalChunks, - expiresAt: new Date(ticket.expiresAt).toISOString(), + expiresAt: upload.expiresAt, next: `Split the file into ${totalChunks} chunk(s) of chunkSizeBytes (the last may be shorter). POST them one at a time, in order, as the raw request body to chunkUrl + "&index=N&isFinal=0" with the returned headers, N counting from 0. Send the last chunk with "&isFinal=1": that request finalizes the recording, so do not call finalize-recording. Retry a failed chunk with the same index. The upload URL stops working once the last chunk is accepted (a retried last chunk then gets 409 saying so; read the clip with get-recording-player-data rather than uploading again) or at expiresAt. Then give the user shareUrl; transcription, and a title when none was given, follow in the background.`, }; }, diff --git a/templates/clips/actions/list-organization-state.test.ts b/templates/clips/actions/list-organization-state.test.ts index 0af076abf39..ea39093dc16 100644 --- a/templates/clips/actions/list-organization-state.test.ts +++ b/templates/clips/actions/list-organization-state.test.ts @@ -28,6 +28,10 @@ vi.mock("../server/lib/agent-recording-access.js", () => ({ isAgentRecordingCaller: () => false, })); +vi.mock("../server/lib/content-uploads.js", () => ({ + uploadFromTrashedPage: () => ({ op: "upload-from-trashed-page" }), +})); + vi.mock("../server/lib/recordings.js", () => ({ getActiveOrganizationId: (...args: unknown[]) => mockGetActiveOrganizationId(...args), @@ -62,6 +66,7 @@ vi.mock("drizzle-orm", () => ({ eq: (...args: unknown[]) => ({ op: "eq", args }), isNotNull: (...args: unknown[]) => ({ op: "isNotNull", args }), isNull: (...args: unknown[]) => ({ op: "isNull", args }), + not: (...args: unknown[]) => ({ op: "not", args }), notExists: (...args: unknown[]) => ({ op: "notExists", args }), or: (...args: unknown[]) => ({ op: "or", args }), sql: () => ({ raw: "sql" }), @@ -85,7 +90,7 @@ function builder( "groupBy", "limit", ]) { - b[method] = () => b; + b[method] = vi.fn(() => b); } b.then = ( resolve: (value: unknown) => unknown, @@ -224,5 +229,12 @@ describe("list-organization-state action", () => { expect(result.folders).toEqual([ expect.objectContaining({ id: "f1", recordingCount: 2 }), ]); + expect(builders[5].where).toHaveBeenCalledWith( + expect.objectContaining({ + args: expect.arrayContaining([ + { op: "not", args: [{ op: "upload-from-trashed-page" }] }, + ]), + }), + ); }); }); diff --git a/templates/clips/actions/list-organization-state.ts b/templates/clips/actions/list-organization-state.ts index b81de36abb2..73b9b6c2977 100644 --- a/templates/clips/actions/list-organization-state.ts +++ b/templates/clips/actions/list-organization-state.ts @@ -13,6 +13,7 @@ import { eq, isNotNull, isNull, + not, notExists, or, sql, @@ -24,6 +25,7 @@ import { agentRecordingAccessFilter, isAgentRecordingCaller, } from "../server/lib/agent-recording-access.js"; +import { uploadFromTrashedPage } from "../server/lib/content-uploads.js"; import { recordingExpiryQuery } from "../server/lib/recording-expiry-query.js"; import { getActiveOrganizationId, @@ -178,6 +180,7 @@ export default defineAction({ recordingExpiryQuery(ownerEmail).available, isNull(schema.recordings.archivedAt), isNull(schema.recordings.trashedAt), + not(uploadFromTrashedPage()), notExists( db .select({ one: sql`1` }) diff --git a/templates/clips/actions/list-recordings.expiry.test.ts b/templates/clips/actions/list-recordings.expiry.test.ts index d5284551921..a463eace66e 100644 --- a/templates/clips/actions/list-recordings.expiry.test.ts +++ b/templates/clips/actions/list-recordings.expiry.test.ts @@ -107,6 +107,7 @@ describe("list recordings expiry with PostgreSQL", () => { schema.recordingComments, schema.recordingReactions, schema.recordingEvents, + schema.recordingContentBindings, schema.meetings, schema.meetingShares, schema.meetingParticipants, diff --git a/templates/clips/actions/list-recordings.test.ts b/templates/clips/actions/list-recordings.test.ts index ea9dec06352..3dcc3d9180d 100644 --- a/templates/clips/actions/list-recordings.test.ts +++ b/templates/clips/actions/list-recordings.test.ts @@ -150,6 +150,10 @@ vi.mock("../server/db/index.js", () => ({ trashedAt: "recordings.trashedAt", }), recordingShares: "recordingShares", + recordingContentBindings: { + recordingId: "recordingContentBindings.recordingId", + origin: "recordingContentBindings.origin", + }, recordingTags: mockTables.recordingTags, recordingViews: Object.assign(mockTables.recordingViews, { recordingId: "recordingViews.recordingId", diff --git a/templates/clips/actions/list-recordings.ts b/templates/clips/actions/list-recordings.ts index 955c4006d3d..6a5a8b234cd 100644 --- a/templates/clips/actions/list-recordings.ts +++ b/templates/clips/actions/list-recordings.ts @@ -23,6 +23,12 @@ import { agentRecordingAccessFilter, isAgentRecordingCaller, } from "../server/lib/agent-recording-access.js"; +import { + activeUploadFromContent, + contentPageUrls, + uploadedFromContent, + uploadFromTrashedPage, +} from "../server/lib/content-uploads.js"; import { resolvePlayerThumbnailUrl } from "../server/lib/player-thumbnail-url.js"; import { resolvePlayerVideoUrl } from "../server/lib/player-video-url.js"; import { recordingExpiryQuery } from "../server/lib/recording-expiry-query.js"; @@ -99,10 +105,18 @@ export function mergeViewCounts( export default defineAction({ description: - "List recordings visible to the current user. Supports filtering by view (library/shared/space/archive/trash/all), folder, space, tag, free-text, and sort. Public/unlisted recordings are discoverable only when owned by or previously viewed by the current user; the shared view returns accessible recordings owned by someone else.", + "List recordings visible to the current user. Supports filtering by view (library/shared/space/archive/trash/all/from-content), folder, space, tag, free-text, and sort. Public/unlisted recordings are discoverable only when owned by or previously viewed by the current user; the shared view returns accessible recordings owned by someone else. The from-content view returns the user's Clips uploaded from Agent-Native Content pages, each with `contentPageUrl`; Library leaves them out.", schema: z.object({ view: z - .enum(["library", "shared", "space", "archive", "trash", "all"]) + .enum([ + "library", + "shared", + "space", + "archive", + "trash", + "all", + "from-content", + ]) .default("library") .describe("Which list to show"), folderId: z @@ -172,6 +186,7 @@ export default defineAction({ userEmail: ctx?.userEmail, }, ), + not(uploadFromTrashedPage()), ]; if (args.recordingIds) { @@ -186,8 +201,11 @@ export default defineAction({ // Library = "Your personal recordings in the active org". `accessFilter` // admits all owner rows regardless of org, so library must add both the - // owner-email and current-org predicates to scope correctly. - if (args.view === "library") { + // owner-email and current-org predicates to scope correctly. A Clip + // uploaded from a Content page is never in Library's own list, and lists + // under From Content, or in a folder or space someone moved it into, only + // while its page isn't in Trash; one linked into a page stays in Library. + if (args.view === "library" || args.view === "from-content") { const email = getRequestUserEmail(); if (email) { whereClauses.push( @@ -197,6 +215,11 @@ export default defineAction({ if (orgId) { whereClauses.push(eq(schema.recordings.organizationId, orgId)); } + if (args.view === "from-content") { + whereClauses.push(activeUploadFromContent()); + } else if (args.folderId == null) { + whereClauses.push(not(uploadedFromContent())); + } } if (args.view === "shared") { @@ -410,10 +433,15 @@ export default defineAction({ .groupBy(schema.recordingAgentViews.recordingId), ]) : Promise.resolve(null); - const [ownerProfiles, tagRows, viewRows] = await Promise.all([ + const contentPagesPromise = + args.view === "from-content" + ? contentPageUrls(ids) + : Promise.resolve(new Map()); + const [ownerProfiles, tagRows, viewRows, contentPages] = await Promise.all([ ownerProfilesPromise, tagRowsPromise, viewRowsPromise, + contentPagesPromise, ]); const tagsByRec: Record = {}; @@ -477,6 +505,9 @@ export default defineAction({ ...resolveListRecordingMedia(r, args.includeMedia), transcriptStatus: row.transcriptStatus ?? null, transcriptHasText: Number(row.transcriptHasText ?? 0) > 0, + ...(args.view === "from-content" + ? { contentPageUrl: contentPages.get(r.id) ?? null } + : {}), }; }); diff --git a/templates/clips/actions/navigate.ts b/templates/clips/actions/navigate.ts index 51653c37c7b..feb596fd70e 100644 --- a/templates/clips/actions/navigate.ts +++ b/templates/clips/actions/navigate.ts @@ -7,6 +7,7 @@ const Views = [ "shared", "spaces", "space", + "from-content", "archive", "trash", "record", diff --git a/templates/clips/actions/view-screen.test.ts b/templates/clips/actions/view-screen.test.ts index dc3661ff695..a0da0b996bd 100644 --- a/templates/clips/actions/view-screen.test.ts +++ b/templates/clips/actions/view-screen.test.ts @@ -178,6 +178,10 @@ vi.mock("../server/db/index.js", () => ({ folders: TABLES.folders, recordings: TABLES.recordings, recordingShares: "recordingShares", + recordingContentBindings: { + recordingId: "bindings.recordingId", + origin: "bindings.origin", + }, recordingViewers: "recordingViewers", recordingComments: "recordingComments", recordingTranscripts: { recordingId: "transcripts.recordingId" }, diff --git a/templates/clips/actions/view-screen.ts b/templates/clips/actions/view-screen.ts index 674bb4391d1..7b9016248f1 100644 --- a/templates/clips/actions/view-screen.ts +++ b/templates/clips/actions/view-screen.ts @@ -21,6 +21,12 @@ import { z } from "zod"; import { getDb, schema } from "../server/db/index.js"; import { agentRecordingAccessFilter } from "../server/lib/agent-recording-access.js"; +import { + activeUploadFromContent, + contentPageUrls, + uploadedFromContent, + uploadFromTrashedPage, +} from "../server/lib/content-uploads.js"; import { loadMeetingRecordingContent } from "../server/lib/meeting-recording-access.js"; import { listingThumbnailUrl } from "../server/lib/player-thumbnail-url.js"; import { recordingExpiryQuery } from "../server/lib/recording-expiry-query.js"; @@ -199,10 +205,13 @@ async function fetchBugReportSummary(recordingId: string) { async function fetchLibrary({ folderId, shared = false, + fromContent = false, organizationId, }: { folderId?: string; shared?: boolean; + /** The caller's uploads from Content pages, which Library leaves out. */ + fromContent?: boolean; organizationId?: string | null; }) { const db = getDb(); @@ -218,6 +227,7 @@ async function fetchLibrary({ ), isNull(schema.recordings.archivedAt), isNull(schema.recordings.trashedAt), + not(uploadFromTrashedPage()), ]; if (shared) { conditions.push( @@ -230,6 +240,8 @@ async function fetchLibrary({ if (organizationId) { conditions.push(eq(schema.recordings.organizationId, organizationId)); } + if (fromContent) conditions.push(activeUploadFromContent()); + else if (!folderId) conditions.push(not(uploadedFromContent())); } const resolvedDb = await Promise.resolve(db); const meetingRecordingIds = resolvedDb @@ -602,6 +614,21 @@ export default defineAction({ }; break; } + case "from-content": { + const recordings = await fetchLibrary({ + fromContent: true, + organizationId, + }); + const pages = await contentPageUrls(recordings.map((r) => r.id)); + screen.fromContent = { + count: recordings.length, + recordings: recordings.map((recording) => ({ + ...recording, + contentPageUrl: pages.get(recording.id) ?? null, + })), + }; + break; + } case "spaces": { screen.spaces = await fetchSpaces(organizationId); break; diff --git a/templates/clips/app/components/library/empty-state.tsx b/templates/clips/app/components/library/empty-state.tsx index a5bc871f44a..bfe9151dfc7 100644 --- a/templates/clips/app/components/library/empty-state.tsx +++ b/templates/clips/app/components/library/empty-state.tsx @@ -6,6 +6,7 @@ import { IconArchive, IconTrash, IconLink, + IconFileText, } from "@tabler/icons-react"; import { type ComponentType, type ReactNode, useState } from "react"; import { Link, useNavigate } from "react-router"; @@ -30,6 +31,7 @@ type EmptyKind = | "space" | "archive" | "trash" + | "fromContent" | "search"; const ICONS: Record> = { @@ -39,6 +41,7 @@ const ICONS: Record> = { space: IconUsersGroup, archive: IconArchive, trash: IconTrash, + fromContent: IconFileText, search: IconVideo, }; @@ -47,6 +50,7 @@ const BACK_TO_LIBRARY_KINDS = new Set([ "shared", "archive", "trash", + "fromContent", ]); interface AppEmptyStateProps { diff --git a/templates/clips/app/components/library/library-grid.tsx b/templates/clips/app/components/library/library-grid.tsx index 7c35c45e174..cad33ddc09f 100644 --- a/templates/clips/app/components/library/library-grid.tsx +++ b/templates/clips/app/components/library/library-grid.tsx @@ -78,12 +78,26 @@ import { SearchBar } from "./search-bar"; import { SortMenu, type SortKey } from "./sort-menu"; interface LibraryGridProps { - view: "library" | "shared" | "space" | "archive" | "trash" | "all"; + view: + | "library" + | "shared" + | "space" + | "archive" + | "trash" + | "all" + | "from-content"; /** "image" narrows the same list to screenshots; defaults to everything. */ kind?: "video" | "image" | "all"; folderId?: string | null; spaceId?: string | null; - emptyKind?: "library" | "shared" | "folder" | "space" | "archive" | "trash"; + emptyKind?: + | "library" + | "shared" + | "folder" + | "space" + | "archive" + | "trash" + | "fromContent"; title?: string; breadcrumbItems?: readonly PageBreadcrumbItem[]; tagFilter?: string | null; diff --git a/templates/clips/app/components/library/library-layout.tsx b/templates/clips/app/components/library/library-layout.tsx index 699e077c170..9134aec760d 100644 --- a/templates/clips/app/components/library/library-layout.tsx +++ b/templates/clips/app/components/library/library-layout.tsx @@ -17,6 +17,7 @@ import { IconInbox, IconPhoto, IconArchive, + IconFileText, IconCalendar, IconMicrophone2, IconTrash, @@ -244,6 +245,9 @@ export function LibraryLayout({ children }: LibraryLayoutProps) { kind: "image", }); const { data: sharedCount } = useRecordingsCount({ view: "shared" }); + const { data: fromContentCount } = useRecordingsCount({ + view: "from-content", + }); const { data: archiveCount } = useRecordingsCount({ view: "archive" }); const { data: trashCount } = useRecordingsCount({ view: "trash" }); const suggestionCountArgs = useMemo(() => { @@ -265,6 +269,9 @@ export function LibraryLayout({ children }: LibraryLayoutProps) { }; } if (location.pathname === "/shared") return { view: "shared" as const }; + if (location.pathname === "/from-content") { + return { view: "from-content" as const }; + } if (location.pathname === "/archive") return { view: "archive" as const }; if (location.pathname === "/trash") return { view: "trash" as const }; if (spaceId && location.pathname.startsWith(`/spaces/${spaceId}`)) { @@ -468,7 +475,7 @@ export function LibraryLayout({ children }: LibraryLayoutProps) { label: string; icon: React.ComponentType<{ className?: string }>; match: (path: string) => boolean; - count?: number; + count?: number | undefined; }[] = [ { to: "/library", @@ -507,6 +514,18 @@ export function LibraryLayout({ children }: LibraryLayoutProps) { p.startsWith("/spaces/") || (p.startsWith("/r/") && recordingSection === "spaces"), }, + // Only for someone who has uploaded from a Content page. + ...(fromContentCount || location.pathname === "/from-content" + ? [ + { + to: "/from-content", + label: t("navigation.fromContent"), + icon: IconFileText, + match: (p: string) => p === "/from-content", + count: fromContentCount, + }, + ] + : []), ...(meetingsLabEnabled ? [ { diff --git a/templates/clips/app/components/library/recording-card.tsx b/templates/clips/app/components/library/recording-card.tsx index 8fbd466bcba..4f6c09e9878 100644 --- a/templates/clips/app/components/library/recording-card.tsx +++ b/templates/clips/app/components/library/recording-card.tsx @@ -703,6 +703,18 @@ export function RecordingCard({ + {recording.contentPageUrl ? ( + + + {t("clipsFinalRaw.openContentPage")} + + ) : null} + {recording.tags.length > 0 && (
{recording.tags.slice(0, 2).map((t) => ( @@ -730,6 +742,18 @@ export function RecordingCard({ {t("clipsFinalRaw.view")} + {recording.contentPageUrl ? ( + + + + {t("clipsFinalRaw.openContentPage")} + + + ) : null} {onShare && ( onShare(recording)}> diff --git a/templates/clips/app/hooks/use-library.ts b/templates/clips/app/hooks/use-library.ts index d4f9a56619e..22f6a7a8b8e 100644 --- a/templates/clips/app/hooks/use-library.ts +++ b/templates/clips/app/hooks/use-library.ts @@ -52,10 +52,19 @@ export interface RecordingSummary { height: number; transcriptStatus?: "pending" | "streaming" | "ready" | "failed" | null; transcriptHasText?: boolean; + /** From Content only: the page the Clip was uploaded to. */ + contentPageUrl?: string | null; } export interface ListRecordingsArgs { - view?: "library" | "shared" | "space" | "archive" | "trash" | "all"; + view?: + | "library" + | "shared" + | "space" + | "archive" + | "trash" + | "all" + | "from-content"; /** "image" is the Screenshots view; omitted means clips and screenshots. */ kind?: "video" | "image" | "all"; folderId?: string | null; diff --git a/templates/clips/app/hooks/use-navigation-state.test.ts b/templates/clips/app/hooks/use-navigation-state.test.ts index e6ba5ee8a18..9dcbaf35982 100644 --- a/templates/clips/app/hooks/use-navigation-state.test.ts +++ b/templates/clips/app/hooks/use-navigation-state.test.ts @@ -11,6 +11,13 @@ describe("Clips shared navigation", () => { expect(pathFromCommand({ view: "shared" })).toBe("/shared"); }); + it("describes and maps the From Content list", () => { + expect(stateFromLocation("/from-content", "")).toEqual({ + view: "from-content", + }); + expect(pathFromCommand({ view: "from-content" })).toBe("/from-content"); + }); + it("describes the focused viewer panel and requested timestamp", () => { expect( stateFromLocation("/r/recording-1", "?panel=transcript&at=42000"), diff --git a/templates/clips/app/hooks/use-navigation-state.ts b/templates/clips/app/hooks/use-navigation-state.ts index 86f7998596a..55226f110d7 100644 --- a/templates/clips/app/hooks/use-navigation-state.ts +++ b/templates/clips/app/hooks/use-navigation-state.ts @@ -13,6 +13,7 @@ export type ClipsView = | "shared" | "spaces" | "space" + | "from-content" | "archive" | "trash" | "record" @@ -156,6 +157,7 @@ export function stateFromLocation( if (p === "/screenshots") return { view: "screenshots" }; if (p === "/spaces") return { view: "spaces" }; if (p === "/shared") return { view: "shared" }; + if (p === "/from-content") return { view: "from-content" }; if (p === "/archive") return { view: "archive" }; if (p === "/trash") return { view: "trash" }; if (p === "/record") return { view: "record" }; @@ -214,6 +216,8 @@ export function pathFromCommand(cmd: NavigateCommand): string { return "/spaces"; case "shared": return "/shared"; + case "from-content": + return "/from-content"; case "archive": return "/archive"; case "trash": diff --git a/templates/clips/app/i18n/ar-SA.ts b/templates/clips/app/i18n/ar-SA.ts index 7ea9a59a109..b240855b50d 100644 --- a/templates/clips/app/i18n/ar-SA.ts +++ b/templates/clips/app/i18n/ar-SA.ts @@ -214,6 +214,7 @@ const messages = { spaces: "المساحات", meetings: "الاجتماعات", dictate: "إملاء", + fromContent: "من Content", archive: "الأرشيف", trash: "المهملات", agent: "الوكيل", @@ -269,6 +270,10 @@ const messages = { title: "لا توجد تسجيلات مؤرشفة", body: "تبقى Clips المؤرشفة هنا حتى تصبح مستعدًا لإعادتها.", }, + fromContent: { + title: "لا توجد عمليات رفع من Content بعد", + body: "تُحفظ هنا مقاطع الفيديو التي ترفعها إلى صفحة في Content بصفتها Clips.", + }, trash: { title: "المهملات فارغة", body: "تبقى Clips المحذوفة هنا لمدة 30 يومًا قبل إزالتها نهائيًا.", @@ -2288,6 +2293,8 @@ const messages = { goHome: "الانتقال إلى الصفحة الرئيسية", testMic: "اختبار الميكروفون", archivePageTitle: "الأرشيف · Clips", + fromContentPageTitle: "من Content · Clips", + openContentPage: "فتح الصفحة", insightsPageTitle: "الرؤى · Clips", folderPageTitle: "المجلد · Clips", spacePageTitle: "المساحة · Clips", diff --git a/templates/clips/app/i18n/de-DE.ts b/templates/clips/app/i18n/de-DE.ts index cf909b2373b..e604714677c 100644 --- a/templates/clips/app/i18n/de-DE.ts +++ b/templates/clips/app/i18n/de-DE.ts @@ -231,6 +231,7 @@ const messages = { spaces: "Bereiche", meetings: "Treffen", dictate: "Diktieren", + fromContent: "Aus Content", archive: "Archiv", trash: "Papierkorb", agent: "Agent", @@ -288,6 +289,10 @@ const messages = { title: "Keine archivierten Aufnahmen", body: "Archivierte Clips bleiben hier, bis du sie zurückholen möchtest.", }, + fromContent: { + title: "Noch keine Uploads aus Content", + body: "Videos, die du auf eine Content-Seite hochlädst, werden hier als Clips gespeichert.", + }, trash: { title: "Der Papierkorb ist leer", body: "Gelöschte Clips bleiben 30 Tage hier, bevor sie endgültig entfernt werden.", @@ -2361,6 +2366,8 @@ const messages = { goHome: "Zur Startseite", testMic: "Mikrofon testen", archivePageTitle: "Archiv · Clips", + fromContentPageTitle: "Aus Content · Clips", + openContentPage: "Seite öffnen", insightsPageTitle: "Einblicke · Clips", folderPageTitle: "Ordner · Clips", spacePageTitle: "Bereich · Clips", diff --git a/templates/clips/app/i18n/en-US.ts b/templates/clips/app/i18n/en-US.ts index ec32d83f06d..efe31878783 100644 --- a/templates/clips/app/i18n/en-US.ts +++ b/templates/clips/app/i18n/en-US.ts @@ -161,6 +161,7 @@ const messages = { spaces: "Spaces", meetings: "Meetings", dictate: "Dictate", + fromContent: "From Content", archive: "Archive", trash: "Trash", agent: "Agent", @@ -216,6 +217,10 @@ const messages = { title: "No archived recordings", body: "Archived Clips stay here until you’re ready to bring them back.", }, + fromContent: { + title: "No uploads from Content yet", + body: "Videos you upload to a Content page are saved here as Clips.", + }, trash: { title: "Trash is empty", body: "Deleted Clips stay here for 30 days before they’re permanently removed.", @@ -2282,6 +2287,8 @@ const messages = { goHome: "Go home", testMic: "Test mic", archivePageTitle: "Archive · Clips", + fromContentPageTitle: "From Content · Clips", + openContentPage: "Open page", insightsPageTitle: "Insights · Clips", folderPageTitle: "Folder · Clips", spacePageTitle: "Space · Clips", diff --git a/templates/clips/app/i18n/es-ES.ts b/templates/clips/app/i18n/es-ES.ts index c945c1d1950..ff7692274d0 100644 --- a/templates/clips/app/i18n/es-ES.ts +++ b/templates/clips/app/i18n/es-ES.ts @@ -227,6 +227,7 @@ const messages = { spaces: "Espacios", meetings: "Reuniones", dictate: "Dictar", + fromContent: "Desde Content", archive: "Archivo", trash: "Papelera", agent: "Agente", @@ -284,6 +285,10 @@ const messages = { title: "No hay grabaciones archivadas", body: "Los Clips archivados se quedan aquí hasta que quieras recuperarlos.", }, + fromContent: { + title: "Aún no hay subidas desde Content", + body: "Los vídeos que subes a una página de Content se guardan aquí como Clips.", + }, trash: { title: "La papelera está vacía", body: "Los Clips eliminados se quedan aquí durante 30 días antes de eliminarse permanentemente.", @@ -2349,6 +2354,8 @@ const messages = { goHome: "Ir al inicio", testMic: "Probar micrófono", archivePageTitle: "Archivo · Clips", + fromContentPageTitle: "Desde Content · Clips", + openContentPage: "Abrir página", insightsPageTitle: "Información · Clips", folderPageTitle: "Carpeta · Clips", spacePageTitle: "Espacio · Clips", diff --git a/templates/clips/app/i18n/fr-FR.ts b/templates/clips/app/i18n/fr-FR.ts index 25ed6e612c8..441d9eca141 100644 --- a/templates/clips/app/i18n/fr-FR.ts +++ b/templates/clips/app/i18n/fr-FR.ts @@ -229,6 +229,7 @@ const messages = { spaces: "Espaces", meetings: "Réunions", dictate: "Dicter", + fromContent: "Depuis Content", archive: "Archiver", trash: "Corbeille", agent: "Agent", @@ -286,6 +287,10 @@ const messages = { title: "Aucun enregistrement archivé", body: "Les Clips archivés restent ici jusqu’à ce que vous souhaitiez les récupérer.", }, + fromContent: { + title: "Aucun envoi depuis Content pour l’instant", + body: "Les vidéos que vous envoyez sur une page Content sont enregistrées ici en tant que Clips.", + }, trash: { title: "La corbeille est vide", body: "Les Clips supprimés restent ici pendant 30 jours avant d’être supprimés définitivement.", @@ -2362,6 +2367,8 @@ const messages = { goHome: "Aller à l’accueil", testMic: "Tester le micro", archivePageTitle: "Archives · Clips", + fromContentPageTitle: "Depuis Content · Clips", + openContentPage: "Ouvrir la page", insightsPageTitle: "Analyses · Clips", folderPageTitle: "Dossier · Clips", spacePageTitle: "Espace · Clips", diff --git a/templates/clips/app/i18n/hi-IN.ts b/templates/clips/app/i18n/hi-IN.ts index b6cb0cc076f..3c4a4be795b 100644 --- a/templates/clips/app/i18n/hi-IN.ts +++ b/templates/clips/app/i18n/hi-IN.ts @@ -217,6 +217,7 @@ const messages = { spaces: "स्पेस", meetings: "मीटिंग्स", dictate: "डिक्टेट", + fromContent: "Content से", archive: "आर्काइव", trash: "ट्रैश", agent: "एजेंट", @@ -272,6 +273,10 @@ const messages = { title: "कोई आर्काइव की गई रिकॉर्डिंग नहीं", body: "आर्काइव किए गए Clips यहाँ तब तक रहते हैं जब तक आप उन्हें वापस लाना न चाहें।", }, + fromContent: { + title: "Content से अभी तक कोई अपलोड नहीं", + body: "Content पेज पर अपलोड किए गए वीडियो यहाँ Clips के रूप में सहेजे जाते हैं।", + }, trash: { title: "ट्रैश खाली है", body: "हटाए गए Clips स्थायी रूप से हटने से पहले 30 दिनों तक यहाँ रहते हैं।", @@ -2267,6 +2272,8 @@ const messages = { goHome: "होम पर जाएं", testMic: "माइक टेस्ट करें", archivePageTitle: "आर्काइव · Clips", + fromContentPageTitle: "Content से · Clips", + openContentPage: "पेज खोलें", insightsPageTitle: "इनसाइट्स · Clips", folderPageTitle: "फ़ोल्डर · Clips", spacePageTitle: "स्पेस · Clips", diff --git a/templates/clips/app/i18n/ja-JP.ts b/templates/clips/app/i18n/ja-JP.ts index 333defddddb..8a96f906752 100644 --- a/templates/clips/app/i18n/ja-JP.ts +++ b/templates/clips/app/i18n/ja-JP.ts @@ -225,6 +225,7 @@ const messages = { spaces: "スペース", meetings: "ミーティング", dictate: "ディクテーション", + fromContent: "Content から", archive: "アーカイブ", trash: "ゴミ箱", agent: "エージェント", @@ -281,6 +282,10 @@ const messages = { title: "アーカイブした録画はありません", body: "アーカイブした Clips は、戻すまでここに保管されます。", }, + fromContent: { + title: "Content からのアップロードはまだありません", + body: "Content のページにアップロードした動画は、Clips としてここに保存されます。", + }, trash: { title: "ゴミ箱は空です", body: "削除した Clips は完全に削除されるまで 30 日間ここに保管されます。", @@ -2330,6 +2335,8 @@ const messages = { goHome: "ホームへ移動", testMic: "マイクをテスト", archivePageTitle: "アーカイブ · Clips", + fromContentPageTitle: "Content から · Clips", + openContentPage: "ページを開く", insightsPageTitle: "インサイト · Clips", folderPageTitle: "フォルダー · Clips", spacePageTitle: "スペース · Clips", diff --git a/templates/clips/app/i18n/ko-KR.ts b/templates/clips/app/i18n/ko-KR.ts index 6db05b58f1e..275b8306fd7 100644 --- a/templates/clips/app/i18n/ko-KR.ts +++ b/templates/clips/app/i18n/ko-KR.ts @@ -221,6 +221,7 @@ const messages = { spaces: "스페이스", meetings: "회의", dictate: "받아쓰기", + fromContent: "Content에서", archive: "보관함", trash: "휴지통", agent: "에이전트", @@ -276,6 +277,10 @@ const messages = { title: "보관한 녹화가 없습니다", body: "보관한 Clips는 다시 가져올 때까지 여기에 보관됩니다.", }, + fromContent: { + title: "아직 Content에서 업로드한 항목이 없습니다", + body: "Content 페이지에 업로드한 동영상은 여기에 Clips로 저장됩니다.", + }, trash: { title: "휴지통이 비어 있습니다", body: "삭제한 Clips는 영구 삭제되기 전 30일 동안 여기에 보관됩니다.", @@ -2293,6 +2298,8 @@ const messages = { goHome: "홈으로 이동", testMic: "마이크 테스트", archivePageTitle: "보관함 · Clips", + fromContentPageTitle: "Content에서 · Clips", + openContentPage: "페이지 열기", insightsPageTitle: "인사이트 · Clips", folderPageTitle: "폴더 · Clips", spacePageTitle: "스페이스 · Clips", diff --git a/templates/clips/app/i18n/pt-BR.ts b/templates/clips/app/i18n/pt-BR.ts index abe7c2dd971..feeb7859262 100644 --- a/templates/clips/app/i18n/pt-BR.ts +++ b/templates/clips/app/i18n/pt-BR.ts @@ -227,6 +227,7 @@ const messages = { spaces: "Espaços", meetings: "Reuniões", dictate: "Ditar", + fromContent: "Do Content", archive: "Arquivo", trash: "Lixeira", agent: "Agente", @@ -284,6 +285,10 @@ const messages = { title: "Nenhuma gravação arquivada", body: "Clips arquivados ficam aqui até você querer trazê-los de volta.", }, + fromContent: { + title: "Nenhum envio do Content ainda", + body: "Os vídeos que você envia para uma página do Content são salvos aqui como Clips.", + }, trash: { title: "A lixeira está vazia", body: "Clips excluídos ficam aqui por 30 dias antes de serem removidos permanentemente.", @@ -2335,6 +2340,8 @@ const messages = { goHome: "Ir para o início", testMic: "Testar microfone", archivePageTitle: "Arquivo · Clips", + fromContentPageTitle: "Do Content · Clips", + openContentPage: "Abrir página", insightsPageTitle: "Informações · Clips", folderPageTitle: "Pasta · Clips", spacePageTitle: "Espaço · Clips", diff --git a/templates/clips/app/i18n/zh-CN.ts b/templates/clips/app/i18n/zh-CN.ts index c255fe55063..9109e27a973 100644 --- a/templates/clips/app/i18n/zh-CN.ts +++ b/templates/clips/app/i18n/zh-CN.ts @@ -208,6 +208,7 @@ const messages = { spaces: "空间", meetings: "会议", dictate: "听写", + fromContent: "来自 Content", archive: "归档", trash: "废纸篓", agent: "代理", @@ -263,6 +264,10 @@ const messages = { title: "暂无归档的录制", body: "归档的 Clips 会一直保存在这里,直到你准备好将它们恢复。", }, + fromContent: { + title: "还没有来自 Content 的上传", + body: "你上传到 Content 页面的视频会作为 Clips 保存在这里。", + }, trash: { title: "废纸篓是空的", body: "删除的 Clips 会在这里保留 30 天,然后永久移除。", @@ -2190,6 +2195,8 @@ const messages = { goHome: "返回首页", testMic: "测试麦克风", archivePageTitle: "归档 · Clips", + fromContentPageTitle: "来自 Content · Clips", + openContentPage: "打开页面", insightsPageTitle: "洞察 · Clips", folderPageTitle: "文件夹 · Clips", spacePageTitle: "空间 · Clips", diff --git a/templates/clips/app/i18n/zh-TW.ts b/templates/clips/app/i18n/zh-TW.ts index 18133d26e02..09988a45381 100644 --- a/templates/clips/app/i18n/zh-TW.ts +++ b/templates/clips/app/i18n/zh-TW.ts @@ -209,6 +209,7 @@ const messages = { spaces: "空間", meetings: "會議", dictate: "聽寫", + fromContent: "來自 Content", archive: "封存", trash: "廢紙簍", agent: "代理", @@ -264,6 +265,10 @@ const messages = { title: "尚無封存的錄製", body: "封存的 Clips 會保存在這裡,直到你準備好將它們恢復。", }, + fromContent: { + title: "還沒有來自 Content 的上傳", + body: "你上傳到 Content 頁面的影片會以 Clips 的形式儲存在這裡。", + }, trash: { title: "廢紙簍是空的", body: "刪除的 Clips 會在這裡保留 30 天,然後永久移除。", @@ -2178,6 +2183,8 @@ const messages = { goHome: "返回首頁", testMic: "測試麥克風", archivePageTitle: "封存 · Clips", + fromContentPageTitle: "來自 Content · Clips", + openContentPage: "開啟頁面", insightsPageTitle: "洞察 · Clips", folderPageTitle: "資料夾 · Clips", spacePageTitle: "空間 · Clips", diff --git a/templates/clips/app/routes/_app.from-content.tsx b/templates/clips/app/routes/_app.from-content.tsx new file mode 100644 index 00000000000..f17b515d41e --- /dev/null +++ b/templates/clips/app/routes/_app.from-content.tsx @@ -0,0 +1,19 @@ +import { useT } from "@agent-native/core/client/i18n"; + +import { LibraryGrid } from "@/components/library/library-grid"; +import enMessages from "@/i18n/en-US"; + +export function meta() { + return [{ title: enMessages.clipsFinalRaw.fromContentPageTitle }]; +} + +export default function FromContentRoute() { + const t = useT(); + return ( + + ); +} diff --git a/templates/clips/server/db/schema.ts b/templates/clips/server/db/schema.ts index 600d78e9af3..1281cda4a74 100644 --- a/templates/clips/server/db/schema.ts +++ b/templates/clips/server/db/schema.ts @@ -374,6 +374,16 @@ export const recordingContentBindings = table( // arrives late cannot undo a newer one. stateChangedAt: text("state_changed_at"), createdBy: text("created_by").notNull(), + // The file an editor picked, from Content: a start repeated with it + // reaches the same upload. Null for a link. + uploadKey: text("upload_key"), + // What the first start said of that file. A repeat must say the same, + // and gets the plan made for it. + uploadMimeType: text("upload_mime_type"), + uploadSizeBytes: bigint("upload_size_bytes", { mode: "number" }), + // When a start of it failed without Clips saying it left nothing: if it + // left no recording, the next repeat starts it again under the same id. + uploadStartFailedAt: text("upload_start_failed_at"), createdAt: text("created_at").notNull().default(now()), updatedAt: text("updated_at").notNull().default(now()), }, @@ -381,6 +391,14 @@ export const recordingContentBindings = table( recordingDocumentUnique: uniqueIndex( "recording_content_bindings_recording_document_unique_idx", ).on(binding.recordingId, binding.contentApp, binding.documentId), + uploadKeyUnique: uniqueIndex( + "recording_content_bindings_upload_key_unique_idx", + ).on( + binding.contentApp, + binding.documentId, + binding.createdBy, + binding.uploadKey, + ), documentIndex: index("recording_content_bindings_document_idx").on( binding.contentApp, binding.documentId, diff --git a/templates/clips/server/lib/content-clip-peer.ts b/templates/clips/server/lib/content-clip-peer.ts index 53370e24812..f58db2f3aab 100644 --- a/templates/clips/server/lib/content-clip-peer.ts +++ b/templates/clips/server/lib/content-clip-peer.ts @@ -82,7 +82,7 @@ interface OpenedClip { binding: ContentClipBinding | null; } -const PRIVATE_HEADERS = { +export const PRIVATE_HEADERS = { "cache-control": "private, no-store", "x-content-type-options": "nosniff", "referrer-policy": "no-referrer", @@ -161,15 +161,25 @@ function isDocumentId(value: unknown): value is string { ); } +/** Who Content vouched for, on which document, before any Clip is named. */ +export interface VerifiedContentPeer { + contentApp: string; + documentId: string; + scope: ContentClipScope; + /** Null for a signed-out reader of a public document. */ + viewerEmail: string | null; + viewerName: string | null; + claims: Record; +} + /** * The reader Content vouched for, or the response refusing the request. Every * refusal is the same bare 401, so a caller learns nothing about the Clip. */ -async function verifyPeer( +export async function verifyContentPeerToken( event: H3Event, - recordingId: string, scopes: readonly ContentClipScope[], -): Promise { +): Promise { const peerSecret = readContentPeerSecret(); if (peerSecret.refused) { return json(peerSecret.refused.body, peerSecret.refused.status); @@ -206,9 +216,6 @@ async function verifyPeer( const contentApp = contentAppFromIssuer(claims.iss); const documentId = claims[CONTENT_CLIP_CLAIMS.documentId]; if (!contentApp || !isDocumentId(documentId)) return unauthorized(); - if (claims[CONTENT_CLIP_CLAIMS.recordingId] !== recordingId) { - return unauthorized(); - } // Content mints a token per call; a long-lived one would outlast the // document access Content checked when it signed it. @@ -239,6 +246,27 @@ async function verifyPeer( ? name.trim().slice(0, MAX_VIEWER_NAME_CHARS) || null : null; + return { + contentApp, + documentId, + scope: claims.scope as ContentClipScope, + viewerEmail, + viewerName, + claims, + }; +} + +async function verifyPeer( + event: H3Event, + recordingId: string, + scopes: readonly ContentClipScope[], +): Promise { + const peer = await verifyContentPeerToken(event, scopes); + if (peer instanceof Response) return peer; + if (peer.claims[CONTENT_CLIP_CLAIMS.recordingId] !== recordingId) { + return unauthorized(); + } + const { contentApp, documentId, viewerEmail, viewerName } = peer; return { contentApp, documentId, recordingId, viewerEmail, viewerName }; } diff --git a/templates/clips/server/lib/content-clip-uploads.test.ts b/templates/clips/server/lib/content-clip-uploads.test.ts new file mode 100644 index 00000000000..6fcad5f5743 --- /dev/null +++ b/templates/clips/server/lib/content-clip-uploads.test.ts @@ -0,0 +1,2286 @@ +import { createRequire } from "node:module"; +import { pathToFileURL } from "node:url"; + +import { getTableConfig, type PgTable } from "drizzle-orm/pg-core"; +import { drizzle, type PgliteDatabase } from "drizzle-orm/pglite"; +import { mockEvent, type H3Event } from "h3"; +import { + afterAll, + afterEach, + beforeEach, + describe, + expect, + it, + vi, +} from "vitest"; + +const CLIPS_URL = "https://clips.example.test"; +const CONTENT_APP = "https://content.example.test"; +const PEER_SECRET = "content-clips-peer-secret-0123456789abcdef"; + +type CreateRecordingRun = + (typeof import("../../actions/create-recording.js"))["default"]["run"]; +type ChunkOwner = Parameters< + (typeof import("../routes/api/uploads/[recordingId]/chunk.post.js"))["handleRecordingChunk"] +>[1]; +type AbortOverride = Parameters< + (typeof import("../routes/api/uploads/[recordingId]/abort.post.js"))["handleAbortRecordingUpload"] +>[1]; +// oxlint-disable-next-line typescript/no-explicit-any -- each case reads the fields of the route, row or list it called +type RouteBody = any; + +function required(value: T | null | undefined, what: string): T { + if (value === null || value === undefined) + throw new Error(`Expected ${what}`); + return value; +} + +const state = vi.hoisted(() => { + vi.stubEnv("APP_URL", "https://clips.example.test"); + vi.stubEnv("A2A_SECRET", "content-clips-test-secret-0123456789abcdef"); + vi.stubEnv( + "CONTENT_CLIPS_PEER_SECRET", + "content-clips-peer-secret-0123456789abcdef", + ); + vi.stubEnv( + "OAUTH_STATE_SECRET", + "content-clips-grant-secret-0123456789abcdef", + ); + return { + db: null as unknown as PgliteDatabase, + storageStatus: {} as Record, + created: [] as Array<{ + args: Parameters[0]; + context: ReturnType< + (typeof import("@agent-native/core/server"))["getRequestContext"] + >; + caller: Parameters[1]; + }>, + chunks: [] as Array<{ owner: unknown }>, + aborts: [] as AbortOverride[], + // Finalization winning the race: the abort finds the file complete. + finalizedFirst: null as null | { + status: "processing" | "ready"; + answer: Record; + httpStatus?: number; + }, + // Runs once just before the next read of a final chunk's claim: the + // final chunk finishing between a reader's reads. + beforeClaimRead: null as null | (() => Promise), + // Runs once just after the next read of where an upload stands: another + // request landing between a repeated start's read and what it does next. + afterPhaseRead: null as null | (() => Promise), + // Runs once inside the next storage check: another request landing + // between an upload start's key lookup and its binding. + duringStorageCheck: null as null | (() => Promise), + // Runs once before create-recording makes its row: another request + // landing between an upload start's binding and its recording. + duringCreate: null as null | (() => Promise), + // The storage the next recording is made in. + rowStorageMode: null as null | "hosted", + // Runs inside the chunk route while it takes a final chunk. + duringFinalChunk: null as null | (() => Promise), + // What the chunk route makes of a final chunk: finalization finishing the + // recording, parking it until storage is connected, or the chunk failing + // with the recording still uploading. + finalChunkOutcome: null as null | "finish" | "park" | "fail", + // Runs once just before the lease reaper fails the expired uploads it + // found: a request landing between the reaper's read and its update. + beforeReaperFails: null as null | (() => Promise), + // Clips' own abort instead of the stand-in below. + realAbort: false, + released: [] as string[], + }; +}); + +vi.mock("@agent-native/core/db", async (importOriginal) => ({ + ...(await importOriginal()), + createGetDb: () => () => state.db, +})); +vi.mock("@agent-native/core/application-state", async (importOriginal) => { + const real = + await importOriginal< + typeof import("@agent-native/core/application-state") + >(); + return { + ...real, + readAppState: async (key: string) => { + if (key.startsWith("recording-upload-final-chunk-")) { + const before = state.beforeClaimRead; + state.beforeClaimRead = null; + await before?.(); + } + return real.readAppState(key); + }, + }; +}); +vi.mock("./final-chunk-claim.js", async (importOriginal) => { + const real = await importOriginal(); + return { + ...real, + readUploadPhase: async ( + ...args: Parameters + ) => { + const standing = await real.readUploadPhase(...args); + const after = state.afterPhaseRead; + state.afterPhaseRead = null; + await after?.(); + return standing; + }, + }; +}); +// create-recording's own storage, analytics and upload-session work is its +// contract, tested with it; here it makes the row as the caller it ran as. +vi.mock("../../actions/create-recording.js", () => ({ + default: { + run: async (...[args, caller]: Parameters) => { + const { getRequestContext } = await import("@agent-native/core/server"); + const { schema } = await import("../db/index.js"); + const context = getRequestContext(); + state.created.push({ args, context, caller }); + const during = state.duringCreate; + state.duringCreate = null; + await during?.(); + await state.db.insert(schema.recordings).values({ + id: required(args.id, "a recording id"), + organizationId: required(context?.orgId, "an org"), + ownerEmail: required(context?.userEmail, "an owner"), + visibility: args.visibility, + title: args.title, + storageMode: state.rowStorageMode, + status: "uploading", + createdAt: "2026-10-24T00:00:00.000Z", + updatedAt: "2026-10-24T00:00:00.000Z", + }); + return { id: args.id, status: "uploading" }; + }, + }, +})); +vi.mock("../routes/api/uploads/[recordingId]/chunk.post.js", () => ({ + handleRecordingChunk: async (event: H3Event, owner: ChunkOwner) => { + state.chunks.push({ owner }); + if (event.url.searchParams.get("isFinal") !== "1") return { ok: true }; + const during = state.duringFinalChunk; + state.duringFinalChunk = null; + await during?.(); + if (state.finalChunkOutcome === "fail") { + const { setResponseStatus } = await import("h3"); + setResponseStatus(event, 502); + return { ok: false, error: "Chunk upload failed (502)" }; + } + if (state.finalChunkOutcome === "park") { + await client.query( + "UPDATE recordings SET failure_reason = 'Connect storage to finish this upload.' WHERE id = $1", + [owner?.recordingId], + ); + const { setResponseStatus } = await import("h3"); + setResponseStatus(event, 202); + return { + ok: true, + finalized: false, + waitingForStorage: true, + status: "waiting_storage", + }; + } + if (state.finalChunkOutcome === "finish") { + await client.query( + "UPDATE recordings SET status = 'ready' WHERE id = $1 AND status = 'uploading'", + [owner?.recordingId], + ); + return { ok: true, finalized: true, status: "ready" }; + } + return { ok: true }; + }, +})); +// Clips' own abort is tested with it; here it fails the row as it would, +// unless a test runs the abort itself. +vi.mock( + "../routes/api/uploads/[recordingId]/abort.post.js", + async (importOriginal) => { + const real = + await importOriginal< + typeof import("../routes/api/uploads/[recordingId]/abort.post.js") + >(); + return { + handleAbortRecordingUpload: async ( + _event: H3Event, + override: AbortOverride, + ) => { + if (state.realAbort) { + state.aborts.push(override); + return real.handleAbortRecordingUpload(_event, override); + } + return fakeAbort(_event, override); + }, + }; + }, +); +async function fakeAbort(_event: H3Event, override: AbortOverride) { + const recordingId = required(override?.recordingId, "the aborted recording"); + const failureCode = override?.body?.["failureCode"]; + const { schema } = await import("../db/index.js"); + const { eq } = await import("drizzle-orm"); + state.aborts.push(override); + if (state.finalizedFirst) { + await state.db + .update(schema.recordings) + .set({ status: state.finalizedFirst.status }) + .where(eq(schema.recordings.id, recordingId)); + if (state.finalizedFirst.httpStatus) { + const { setResponseStatus } = await import("h3"); + setResponseStatus(_event, state.finalizedFirst.httpStatus); + } + return state.finalizedFirst.answer; + } + await state.db + .update(schema.recordings) + .set({ + status: "failed", + failureCode: typeof failureCode === "string" ? failureCode : null, + }) + .where(eq(schema.recordings.id, recordingId)); + return { ok: true, recordingId, chunksCleared: 0 }; +} +vi.mock("./video-storage.js", async (importOriginal) => ({ + ...(await importOriginal()), + getClipsVideoStorageStatus: async () => { + const during = state.duringStorageCheck; + state.duringStorageCheck = null; + await during?.(); + return state.storageStatus; + }, + releaseHostedClipRecording: async ({ + recordingId, + }: { + recordingId: string; + }) => { + state.released.push(recordingId); + return { ok: true }; + }, +})); + +const coreRequire = createRequire( + new URL("../../../../packages/core/package.json", import.meta.url), +); +const { PGlite } = coreRequire("@electric-sql/pglite"); +const jose = await import(pathToFileURL(coreRequire.resolve("jose")).href); + +const { organizations, orgMembers } = await import("@agent-native/core/org"); +const { withDbExec } = await import("@agent-native/core/db"); +const { schema } = await import("../db/index.js"); +const { CONTENT_CLIP_SCOPES: SCOPES, CONTENT_CLIP_PUBLIC_READER } = + await import("../../shared/content-clip-contract.js"); +function pairedOrigin(): string { + return required(pairedContentOrigin(), "a paired Content origin"); +} +function firstCreated() { + return required(state.created[0], "a create-recording call"); +} +function createdOrgId(): string { + return required( + firstCreated().caller?.orgId, + "the org create-recording ran in", + ); +} + +const { contentClipStorageFor, pairedContentOrigin } = + await import("./content-clip-uploads.js"); +const { contentPageUrls, refuseContentUploadChunk } = + await import("./content-uploads.js"); +const { default: uploadStartRoute } = + await import("../routes/api/content-clips/uploads.post.js"); +const { default: uploadStatusRoute } = + await import("../routes/api/content-clips/upload-status.get.js"); +const { default: contentChunkRoute } = + await import("../routes/api/uploads/[recordingId]/content-chunk.post.js"); +const { default: contentCancelRoute } = + await import("../routes/api/uploads/[recordingId]/content-cancel.post.js"); +const { default: authMiddleware } = await import("../middleware/auth.js"); +const { default: playbackRoute } = + await import("../routes/api/content-clips/[recordingId]/playback.get.js"); +const { createRecordingUploadTicket } = + await import("./recording-upload-ticket.js"); +const { runWithRequestContext } = await import("@agent-native/core/server"); +const { writeAppState } = await import("@agent-native/core/application-state"); +const { reapExpiredUploads } = await import("./upload-lease.js"); +const { default: listRecordings } = + await import("../../actions/list-recordings.js"); + +const EDITOR = "editor@content.example.test"; +const STRANGER = "no-clips-account@content.example.test"; + +let client: Awaited>; + +function createTableSql(table: PgTable): string { + const config = getTableConfig(table); + const columns = config.columns.map((column) => { + const parts = [`"${column.name}" ${column.getSQLType()}`]; + if (column.primary) parts.push("PRIMARY KEY"); + const fallback = column.default; + if (typeof fallback === "string") { + parts.push(`DEFAULT '${fallback.replace(/'/g, "''")}'`); + } else if (typeof fallback === "number" || typeof fallback === "boolean") { + parts.push(`DEFAULT ${fallback}`); + } + return parts.join(" "); + }); + const uniqueIndexes = config.indexes + .filter((index) => index.config.unique) + .map( + (index) => + `CREATE UNIQUE INDEX IF NOT EXISTS "${index.config.name}" ON "${config.name}" (${index.config.columns + .map((column) => `"${(column as { name: string }).name}"`) + .join(", ")});`, + ); + return [ + `CREATE TABLE IF NOT EXISTS "${config.name}" (${columns.join(", ")});`, + ...uniqueIndexes, + ].join("\n"); +} + +function execOn(target: Pick) { + return { + async execute(statement: string | { sql: string; args?: unknown[] }) { + const query = + typeof statement === "string" ? { sql: statement } : statement; + if (/^\s*UPDATE recordings\s+SET status = 'failed'/.test(query.sql)) { + const before = state.beforeReaperFails; + state.beforeReaperFails = null; + await before?.(); + } + let param = 0; + const result = await target.query( + query.sql.replace(/\?/g, () => `$${++param}`), + query.args ?? [], + ); + return { rows: result.rows, rowsAffected: result.affectedRows ?? 0 }; + }, + }; +} + +/** Core reads through its own handle; point it at this test's database. */ +function coreDb() { + return { + ...execOn(client), + transaction: (run: (tx: ReturnType) => Promise) => + client.transaction((tx: Pick) => run(execOn(tx))), + }; +} + +let editors = 0; + +/** + * A new editor with a Clips account in a workspace of their own. Core caches + * memberships per email, so no editor carries over into another test. + */ +async function clipsEditor(): Promise { + const email = `editor-${++editors}@content.example.test`; + const orgId = `org-${editors}`; + await state.db.insert(organizations).values({ + id: orgId, + name: orgId, + createdBy: email, + createdAt: 1, + }); + await state.db.insert(orgMembers).values({ + id: `member-${orgId}`, + orgId, + email, + role: "owner", + joinedAt: 1, + }); + await client.query( + `INSERT INTO "user" (id, email, email_verified) VALUES ($1, $2, true)`, + [`user-${editors}`, email], + ); + return email; +} + +interface TokenInput { + scope?: string; + sub?: string; + documentId?: string; + recordingId?: string; + audience?: string; + issuer?: string; + secret?: string; + extra?: Record; +} + +async function token(input: TokenInput = {}): Promise { + const now = Math.floor(Date.now() / 1000); + return new jose.SignJWT({ + scope: input.scope ?? SCOPES.upload, + document_id: input.documentId ?? "doc-1", + ...(input.recordingId ? { recording_id: input.recordingId } : {}), + sub: input.sub ?? EDITOR, + ...input.extra, + }) + .setProtectedHeader({ alg: "HS256" }) + .setIssuedAt(now) + .setExpirationTime(now + 60) + .setIssuer(input.issuer ?? CONTENT_APP) + .setAudience(input.audience ?? `${CLIPS_URL}/content-clips`) + .sign(new TextEncoder().encode(input.secret ?? PEER_SECRET)); +} + +async function respond(handler: (event: H3Event) => unknown, event: H3Event) { + const result = await withDbExec(coreDb(), () => handler(event)); + if (result instanceof Response) { + return { + status: result.status, + body: await result.clone().json(), + headers: result.headers, + }; + } + return { + status: event.res.status ?? 200, + body: result as RouteBody, + headers: event.res.headers as Headers, + }; +} + +const START = { + fileName: "Quarterly demo.mp4", + sizeBytes: 5 * 1024 * 1024, + mimeType: "video/mp4", + blockId: "block-video-1", +}; + +let uploadKeys = 0; + +/** A start for a newly picked file, unless the body names the pick's key. */ +function startUpload(bearer: string, body: unknown = START) { + if (typeof body === "object" && body !== null && !("uploadKey" in body)) { + body = { + ...body, + uploadKey: `picked-file-${String(++uploadKeys).padStart(8, "0")}`, + }; + } + return respond( + uploadStartRoute, + mockEvent(`${CLIPS_URL}/api/content-clips/uploads`, { + method: "POST", + headers: { + authorization: `Bearer ${bearer}`, + "content-type": "application/json", + }, + body: typeof body === "string" ? body : JSON.stringify(body), + }), + ); +} + +function uploadStatus(bearer: string) { + return respond( + uploadStatusRoute, + mockEvent(`${CLIPS_URL}/api/content-clips/upload-status`, { + headers: { authorization: `Bearer ${bearer}` }, + }), + ); +} + +function postChunk( + url: string, + options: { origin?: string; guardHeaders?: Record } = {}, +) { + const event = mockEvent(url, { + method: "POST", + headers: options.origin ? { origin: options.origin } : {}, + body: Buffer.from("fake video bytes for one chunk"), + }); + event.context.params = { + recordingId: decodeURIComponent( + /\/api\/uploads\/([^/]+)\//.exec(new URL(url).pathname)?.[1] ?? "", + ), + }; + // What the auth guard sets on every request with an Origin it knows. + for (const [name, value] of Object.entries(options.guardHeaders ?? {})) { + event.res.headers.set(name, value); + } + return respond(contentChunkRoute, event); +} + +function postCancel(url: string, origin?: string) { + const event = mockEvent(url, { + method: "POST", + headers: origin ? { origin } : {}, + }); + event.context.params = { + recordingId: decodeURIComponent( + /\/api\/uploads\/([^/]+)\//.exec(new URL(url).pathname)?.[1] ?? "", + ), + }; + return respond(contentCancelRoute, event); +} + +async function rows(table: string) { + return (await client.query(`SELECT * FROM ${table}`)).rows as RouteBody[]; +} + +/** Application state as the editor's own requests read and write it. */ +async function asEditor(editor: string, run: () => Promise): Promise { + return await withDbExec(coreDb(), () => + runWithRequestContext({ userEmail: editor }, run), + ); +} + +beforeEach(async () => { + client = await PGlite.create("memory://"); + state.db = drizzle(client); + // A local development Clips: chunks are buffered in SQL. + state.storageStatus = { + mode: "none", + configured: false, + activeProvider: null, + localSqlChunkScratchAvailable: true, + hostedRemaining: 0, + hostedLimit: 0, + maxClipBytes: 250 * 1024 * 1024, + denyReason: "disabled", + }; + state.created = []; + state.chunks = []; + state.aborts = []; + state.finalizedFirst = null; + state.beforeClaimRead = null; + state.afterPhaseRead = null; + state.duringStorageCheck = null; + state.duringCreate = null; + state.rowStorageMode = null; + state.duringFinalChunk = null; + state.finalChunkOutcome = null; + state.realAbort = false; + state.beforeReaperFails = null; + state.released = []; + for (const table of [ + schema.recordings, + schema.recordingShares, + schema.recordingContentBindings, + schema.recordingContextItems, + schema.recordingViewers, + schema.recordingViews, + schema.recordingAgentViews, + schema.recordingTags, + schema.recordingTranscripts, + schema.meetings, + schema.hostedClipAllowances, + organizations, + orgMembers, + ]) { + await client.exec(createTableSql(table as PgTable)); + } + await client.exec(` + ALTER TABLE organizations + ADD COLUMN a2a_member_delegation_enabled BOOLEAN NOT NULL DEFAULT false; + CREATE TABLE settings ( + key TEXT PRIMARY KEY, value TEXT NOT NULL, updated_at BIGINT NOT NULL + ); + CREATE TABLE "user" ( + id TEXT PRIMARY KEY, email TEXT NOT NULL, email_verified BOOLEAN + ); + `); +}); + +afterEach(async () => { + await client.close(); +}); + +afterAll(() => { + vi.unstubAllEnvs(); +}); + +describe("an editor's upload from a Content page", () => { + it("makes one private recording owned by the editor, bound to the page as an upload", async () => { + const editor = await clipsEditor(); + const result = await startUpload(await token({ sub: editor })); + + expect(result.status).toBe(200); + const [recording] = await rows("recordings"); + expect(await rows("recordings")).toHaveLength(1); + expect(recording).toMatchObject({ + id: result.body.recordingId, + owner_email: editor, + visibility: "private", + status: "uploading", + title: "Quarterly demo", + }); + expect(state.created).toEqual([ + expect.objectContaining({ + args: expect.objectContaining({ + id: result.body.recordingId, + visibility: "private", + mimeType: "video/mp4", + sizeBytes: START.sizeBytes, + recordingSource: "file_upload", + }), + context: expect.objectContaining({ + userEmail: editor, + orgId: createdOrgId(), + authUserId: expect.stringMatching(/^user-/), + }), + }), + ]); + expect(createdOrgId()).toMatch(/^org-/); + expect(await rows("recording_shares")).toEqual([]); + expect(await rows("recording_content_bindings")).toEqual([ + expect.objectContaining({ + recording_id: result.body.recordingId, + content_app: CONTENT_APP, + document_id: "doc-1", + block_id: "block-video-1", + origin: "upload", + state: "active", + created_by: editor, + }), + ]); + }); + + it("returns a chunk URL on Clips whose ticket is the only authority, and no headers to send", async () => { + const editor = await clipsEditor(); + const result = await startUpload(await token({ sub: editor })); + + const chunkUrl = new URL(result.body.chunkUrl); + expect(chunkUrl.origin).toBe(CLIPS_URL); + expect(chunkUrl.pathname).toBe( + `/api/uploads/${result.body.recordingId}/content-chunk`, + ); + expect(chunkUrl.searchParams.get("ticket")).toMatch(/^clipsup_/); + expect(chunkUrl.searchParams.get("mimeType")).toBe("video/mp4"); + const cancelUrl = new URL(result.body.cancelUrl); + expect(cancelUrl.origin).toBe(CLIPS_URL); + expect(cancelUrl.pathname).toBe( + `/api/uploads/${result.body.recordingId}/content-cancel`, + ); + expect([...cancelUrl.searchParams.keys()]).toEqual(["ticket"]); + expect(cancelUrl.searchParams.get("ticket")).toBe( + chunkUrl.searchParams.get("ticket"), + ); + expect(result.body).toMatchObject({ + totalChunks: 2, + chunkSizeBytes: expect.any(Number), + }); + expect(result.body).not.toHaveProperty("headers"); + expect(Date.parse(result.body.expiresAt)).toBeGreaterThan(Date.now()); + }); + + it("lets the page's readers watch the upload but not its other pages", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + const playback = async (documentId: string) => { + const bearer = await token({ + scope: SCOPES.read, + sub: "reader@content.example.test", + documentId, + recordingId: body.recordingId, + }); + const event = mockEvent( + `${CLIPS_URL}/api/content-clips/${body.recordingId}/playback`, + { headers: { authorization: `Bearer ${bearer}` } }, + ); + event.context.params = { recordingId: body.recordingId }; + return respond(playbackRoute, event); + }; + + const onPage = await playback("doc-1"); + expect(onPage.status).toBe(200); + expect(onPage.body.recording.status).toBe("uploading"); + expect((await playback("doc-2")).status).toBe(404); + }); + + it("refuses any token that isn't an editor's upload token", async () => { + const editor = await clipsEditor(); + for (const bearer of [ + await token({ sub: editor, scope: SCOPES.read }), + await token({ sub: editor, scope: SCOPES.comment }), + await token({ sub: editor, recordingId: "rec-1" }), + await token({ sub: CONTENT_CLIP_PUBLIC_READER }), + await token({ sub: editor, secret: "another-secret-0123456789abcdefgh" }), + await token({ sub: editor, audience: `${CLIPS_URL}/other` }), + ]) { + expect((await startUpload(bearer)).status).toBe(401); + expect((await uploadStatus(bearer)).status).toBe(401); + } + expect(await rows("recordings")).toEqual([]); + expect(await rows("recording_content_bindings")).toEqual([]); + }); + + it("never opens a Clip for playback with an upload token", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + const event = mockEvent( + `${CLIPS_URL}/api/content-clips/${body.recordingId}/playback`, + { + headers: { + authorization: `Bearer ${await token({ + sub: editor, + recordingId: body.recordingId, + })}`, + }, + }, + ); + event.context.params = { recordingId: body.recordingId }; + expect((await respond(playbackRoute, event)).status).toBe(401); + }); + + it("refuses a file that isn't a video Clips takes", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + for (const body of [ + { ...START, mimeType: "image/png" }, + { ...START, sizeBytes: 0 }, + { ...START, blockId: "has spaces" }, + { ...START, extra: true }, + { ...START, uploadKey: "short" }, + "{not json", + ]) { + expect((await startUpload(bearer, body)).status).toBe(400); + } + expect( + (await startUpload(bearer, { ...START, fileName: "a".repeat(20_000) })) + .status, + ).toBe(413); + expect(await rows("recordings")).toEqual([]); + }); + + it("says an email with no Clips account needs one, and writes nothing", async () => { + const bearer = await token({ sub: STRANGER }); + + expect((await uploadStatus(bearer)).body).toEqual({ + storage: { state: "account_required" }, + }); + const result = await startUpload(bearer); + expect(result.status).toBe(409); + expect(result.body).toMatchObject({ + code: "storage_required", + storage: { state: "account_required" }, + }); + expect(await rows("recordings")).toEqual([]); + expect(await rows("recording_content_bindings")).toEqual([]); + expect(await rows("organizations")).toEqual([]); + }); + + it("says an account with no workspace yet needs one, without making one", async () => { + await client.query( + `INSERT INTO "user" (id, email, email_verified) VALUES ('user-new', $1, true)`, + [STRANGER], + ); + const result = await uploadStatus(await token({ sub: STRANGER })); + expect(result.body).toEqual({ storage: { state: "account_required" } }); + expect(await rows("organizations")).toEqual([]); + expect(await rows("org_members")).toEqual([]); + }); + + it("refuses the upload with the storage state when the free clips are used up", async () => { + const editor = await clipsEditor(); + state.storageStatus = { + mode: "none", + configured: false, + activeProvider: null, + localSqlChunkScratchAvailable: false, + hostedRemaining: 0, + hostedLimit: 3, + maxClipBytes: 250 * 1024 * 1024, + denyReason: "exhausted", + }; + const bearer = await token({ sub: editor }); + + expect((await uploadStatus(bearer)).body).toEqual({ + storage: { state: "used_up", limit: 3 }, + }); + const result = await startUpload(bearer); + expect(result.status).toBe(409); + expect(result.body.storage).toEqual({ state: "used_up", limit: 3 }); + expect(await rows("recordings")).toEqual([]); + expect(await rows("recording_content_bindings")).toEqual([]); + }); + + it("refuses a file larger than the free storage takes, before anything is made", async () => { + const editor = await clipsEditor(); + state.storageStatus = { + mode: "hosted", + configured: true, + activeProvider: null, + localSqlChunkScratchAvailable: false, + hostedRemaining: 2, + hostedLimit: 3, + maxClipBytes: 1024 * 1024, + }; + const result = await startUpload(await token({ sub: editor })); + expect(result.status).toBe(413); + expect(result.body).toMatchObject({ + code: "upload_too_large", + maxBytes: 1024 * 1024, + }); + expect(await rows("recordings")).toEqual([]); + }); +}); + +describe("a start Content repeats for one picked file", () => { + const KEY = "picked-file-repeated"; + + async function startTwice( + again: Record = {}, + between?: () => Promise, + ) { + const editor = await clipsEditor(); + const first = await startUpload(await token({ sub: editor }), { + ...START, + uploadKey: KEY, + }); + await between?.(); + const repeat = await startUpload(await token({ sub: editor }), { + ...START, + uploadKey: KEY, + ...again, + }); + return { editor, first, repeat }; + } + + it("answers with the same recording, the first start's plan and a ticket of its own, and makes and binds nothing more", async () => { + const { first, repeat } = await startTwice(); + + expect(repeat.status).toBe(200); + expect(repeat.body).toMatchObject({ + recordingId: first.body.recordingId, + totalChunks: first.body.totalChunks, + chunkSizeBytes: first.body.chunkSizeBytes, + }); + const plan = (url: string) => { + const query = new URL(url).searchParams; + query.delete("ticket"); + return query.toString(); + }; + expect(plan(repeat.body.chunkUrl)).toBe(plan(first.body.chunkUrl)); + expect(state.created).toHaveLength(1); + expect(await rows("recordings")).toHaveLength(1); + expect(await rows("recording_content_bindings")).toEqual([ + expect.objectContaining({ + recording_id: first.body.recordingId, + upload_key: KEY, + }), + ]); + // The earlier answer still uploads: a page may be acting on either. + const paired = pairedOrigin(); + const chunk = (url: string) => + postChunk(`${url}&index=0&isFinal=0`, { origin: paired }); + expect((await chunk(repeat.body.chunkUrl)).status).toBe(200); + expect((await chunk(first.body.chunkUrl)).status).toBe(200); + }); + + it("gives two overlapping repeats tickets that both upload and cancel", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const first = await startUpload(bearer, { ...START, uploadKey: KEY }); + const repeats = await Promise.all([ + startUpload(bearer, { ...START, uploadKey: KEY }), + startUpload(bearer, { ...START, uploadKey: KEY }), + ]); + + const paired = pairedOrigin(); + for (const repeat of repeats) { + expect(repeat.status).toBe(200); + expect(repeat.body.recordingId).toBe(first.body.recordingId); + const chunk = await postChunk( + `${repeat.body.chunkUrl}&index=0&isFinal=0`, + { origin: paired }, + ); + expect(chunk.status).toBe(200); + } + for (const repeat of repeats) { + expect(await postCancel(repeat.body.cancelUrl, paired)).toMatchObject({ + status: 200, + body: { outcome: "cancelled", recordingId: first.body.recordingId }, + }); + } + }); + + it("keeps a recording's four latest tickets live, retiring older ones", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const answers = [await startUpload(bearer, { ...START, uploadKey: KEY })]; + for (let repeat = 0; repeat < 4; repeat += 1) { + answers.push(await startUpload(bearer, { ...START, uploadKey: KEY })); + } + + const paired = pairedOrigin(); + const statuses: number[] = []; + for (const answer of answers) { + const chunk = await postChunk( + `${answer.body.chunkUrl}&index=0&isFinal=0`, + { origin: paired }, + ); + statuses.push(chunk.status); + } + expect(statuses).toEqual([401, 200, 200, 200, 200]); + const [ticketRow] = await rows("settings"); + expect(Object.keys(JSON.parse(ticketRow.value).tickets)).toEqual([ + required(answers[0], "a first answer").body.recordingId, + ]); + }); + + it.each([ + ["size", { sizeBytes: START.sizeBytes + 1 }], + ["type", { mimeType: "video/webm" }], + ])( + "refuses a repeat for a file of another %s, and changes nothing", + async (_what, again) => { + let before: unknown; + const { first, repeat } = await startTwice(again, async () => { + before = { + recordings: await rows("recordings"), + bindings: await rows("recording_content_bindings"), + tickets: await rows("settings"), + }; + }); + + expect(repeat.status).toBe(409); + expect(repeat.body).toEqual({ + error: "upload_key_used", + code: "upload_key_used", + }); + expect({ + recordings: await rows("recordings"), + bindings: await rows("recording_content_bindings"), + tickets: await rows("settings"), + }).toEqual(before); + const chunk = await postChunk( + `${first.body.chunkUrl}&index=0&isFinal=0`, + { origin: pairedOrigin() }, + ); + expect(chunk.status).toBe(200); + }, + ); + + it("asks a repeat that lands before the first start's recording exists to try again", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + let early: Awaited> | undefined; + state.duringCreate = async () => { + early = await startUpload(bearer, { ...START, uploadKey: KEY }); + }; + const first = await startUpload(bearer, { ...START, uploadKey: KEY }); + + expect(early).toMatchObject({ + status: 409, + body: { error: "upload_starting", code: "upload_starting" }, + }); + expect(first.status).toBe(200); + expect(state.created).toHaveLength(1); + const again = await startUpload(bearer, { ...START, uploadKey: KEY }); + expect(again.status).toBe(200); + expect(again.body.recordingId).toBe(first.body.recordingId); + }); + + /** create-recording failing as it does when it can't say what it left. */ + async function failCreate(failure: "unconfirmed" | "confirmed" | "untyped") { + if (failure === "untyped") throw new Error("connection reset"); + const { fail } = await import("@agent-native/core/action"); + const cleanupConfirmed = failure === "confirmed"; + fail("Recording could not be created. Refresh and try again.", { + errorCode: "recording_create_failed", + statusCode: 503, + details: { retryable: cleanupConfirmed, cleanupConfirmed }, + }); + } + + async function failedStart(failure: "unconfirmed" | "confirmed" | "untyped") { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + state.duringCreate = () => failCreate(failure); + await expect( + startUpload(bearer, { ...START, uploadKey: KEY }), + ).rejects.toThrow(); + return { editor, bearer }; + } + + it("makes the next start of a file the same recording when its first start couldn't say it left nothing, so one free clip is held", async () => { + const { bearer } = await failedStart("unconfirmed"); + const [kept] = await rows("recording_content_bindings"); + expect(kept).toMatchObject({ upload_key: KEY }); + expect(kept.upload_start_failed_at).not.toBeNull(); + + const retake = await startUpload(bearer, { ...START, uploadKey: KEY }); + + expect(retake.status).toBe(200); + expect(retake.body.recordingId).toBe(kept.recording_id); + // Clips reserves a free clip per recording id: both starts share one. + expect(state.created.map(({ args }) => args.id)).toEqual([ + kept.recording_id, + kept.recording_id, + ]); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ id: kept.recording_id }), + ]); + expect(await rows("recording_content_bindings")).toEqual([ + expect.objectContaining({ + recording_id: kept.recording_id, + upload_start_failed_at: null, + }), + ]); + }); + + it.each(["unconfirmed", "untyped"] as const)( + "keeps the binding of a start that failed %s", + async (failure) => { + await failedStart(failure); + expect(await rows("recording_content_bindings")).toEqual([ + expect.objectContaining({ + upload_key: KEY, + upload_start_failed_at: expect.any(String), + }), + ]); + }, + ); + + it("drops the binding of a start Clips says left nothing, so the next start makes its own recording", async () => { + const { bearer } = await failedStart("confirmed"); + expect(await rows("recording_content_bindings")).toEqual([]); + + const next = await startUpload(bearer, { ...START, uploadKey: KEY }); + expect(next.status).toBe(200); + expect(next.body.recordingId).not.toBe(firstCreated().args.id); + }); + + it("refuses the key of a start that failed after making its recording, and makes nothing more", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + // create-recording made the row and failed it, then couldn't say so. + state.duringCreate = async () => { + const { args, context } = required( + state.created[state.created.length - 1], + "a create-recording call", + ); + await state.db.insert(schema.recordings).values({ + id: required(args.id, "a recording id"), + organizationId: required(context?.orgId, "an org"), + ownerEmail: editor, + visibility: args.visibility, + title: args.title, + status: "failed", + createdAt: "2026-10-24T00:00:00.000Z", + updatedAt: "2026-10-24T00:00:00.000Z", + }); + await failCreate("unconfirmed"); + }; + await expect( + startUpload(bearer, { ...START, uploadKey: KEY }), + ).rejects.toThrow(); + const [kept] = await rows("recording_content_bindings"); + expect(kept.upload_start_failed_at).not.toBeNull(); + + const repeat = await startUpload(bearer, { ...START, uploadKey: KEY }); + + expect(repeat).toMatchObject({ + status: 409, + body: { code: "upload_key_used" }, + }); + expect(state.created).toHaveLength(1); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ id: kept.recording_id, status: "failed" }), + ]); + }); + + it("runs one retake of a failed start when two repeats overlap", async () => { + const { bearer } = await failedStart("unconfirmed"); + const [failedId] = state.created.map(({ args }) => args.id); + // Both read the failed start before either takes it over. + let overlapping: Awaited> | undefined; + state.afterPhaseRead = async () => { + overlapping = await startUpload(bearer, { ...START, uploadKey: KEY }); + }; + + const outer = await startUpload(bearer, { ...START, uploadKey: KEY }); + + expect(overlapping).toMatchObject({ + status: 200, + body: { recordingId: failedId }, + }); + expect(outer).toMatchObject({ + status: 409, + body: { code: "upload_starting" }, + }); + expect(state.created.map(({ args }) => args.id)).toEqual([ + failedId, + failedId, + ]); + expect(await rows("recordings")).toHaveLength(1); + }); + + it("takes over a key whose binding never got its recording, once that's stale, with the same recording", async () => { + const editor = await clipsEditor(); + const longAgo = new Date(Date.now() - 11 * 60 * 1000).toISOString(); + await state.db.insert(schema.recordingContentBindings).values({ + id: "binding-stranded", + recordingId: "rec-never-made", + contentApp: CONTENT_APP, + documentId: "doc-1", + blockId: START.blockId, + origin: "upload", + state: "active", + stateChangedAt: longAgo, + createdBy: editor, + uploadKey: KEY, + uploadMimeType: START.mimeType, + uploadSizeBytes: START.sizeBytes, + createdAt: longAgo, + updatedAt: longAgo, + }); + + const repeat = await startUpload(await token({ sub: editor }), { + ...START, + uploadKey: KEY, + }); + + expect(repeat.status).toBe(200); + expect(repeat.body.recordingId).toBe("rec-never-made"); + expect(state.created.map(({ args }) => args.id)).toEqual([ + "rec-never-made", + ]); + }); + + it.each(["confirmed", "unconfirmed"] as const)( + "leaves the binding of a repeat that took over a slow start, when the slow start then fails %s", + async (failure) => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + let retake: Awaited> | undefined; + state.duringCreate = async () => { + const longAgo = new Date(Date.now() - 11 * 60 * 1000).toISOString(); + await state.db + .update(schema.recordingContentBindings) + .set({ updatedAt: longAgo }); + retake = await startUpload(bearer, { ...START, uploadKey: KEY }); + await failCreate(failure); + }; + + await expect( + startUpload(bearer, { ...START, uploadKey: KEY }), + ).rejects.toThrow(); + + expect(retake?.status).toBe(200); + const chunk = await postChunk( + `${retake?.body.chunkUrl}&index=0&isFinal=0`, + { origin: pairedOrigin() }, + ); + expect(chunk.status).toBe(200); + expect(await rows("recording_content_bindings")).toEqual([ + expect.objectContaining({ + recording_id: retake?.body.recordingId, + upload_start_failed_at: null, + }), + ]); + }, + ); + + it("answers even when the first start took the last free clip", async () => { + const { first, repeat } = await startTwice({}, async () => { + state.storageStatus = { + mode: "none", + configured: false, + activeProvider: null, + localSqlChunkScratchAvailable: false, + hostedRemaining: 0, + hostedLimit: 3, + maxClipBytes: 250 * 1024 * 1024, + denyReason: "exhausted", + }; + }); + + expect(repeat.status).toBe(200); + expect(repeat.body.recordingId).toBe(first.body.recordingId); + }); + + it.each(["processing", "ready"])( + "says the upload already finished once its recording is %s", + async (status) => { + const { first, repeat } = await startTwice({}, () => + client.query("UPDATE recordings SET status = $1", [status]), + ); + + expect(repeat.status).toBe(409); + expect(repeat.body).toEqual({ + error: "upload_already_finished", + code: "upload_already_finished", + recordingId: first.body.recordingId, + status, + }); + expect(await rows("recordings")).toHaveLength(1); + }, + ); + + it.each([ + [ + "once its upload was cancelled", + {}, + () => client.query("UPDATE recordings SET status = 'failed'"), + ], + ["from another block", { blockId: "block-video-2" }, undefined], + ])("refuses the key %s", async (_when, again, between) => { + const { repeat } = await startTwice(again, between); + + expect(repeat.status).toBe(409); + expect(repeat.body).toEqual({ + error: "upload_key_used", + code: "upload_key_used", + }); + expect(await rows("recordings")).toHaveLength(1); + expect(await rows("recording_content_bindings")).toHaveLength(1); + }); + + it("makes a new upload for another key", async () => { + const { first, repeat } = await startTwice({ + uploadKey: "picked-file-another", + }); + + expect(repeat.status).toBe(200); + expect(repeat.body.recordingId).not.toBe(first.body.recordingId); + expect(await rows("recordings")).toHaveLength(2); + expect(await rows("recording_content_bindings")).toHaveLength(2); + }); + + it("never answers another uploader's key with the first one's recording", async () => { + const one = await clipsEditor(); + const two = await clipsEditor(); + const first = await startUpload(await token({ sub: one }), { + ...START, + uploadKey: KEY, + }); + const second = await startUpload(await token({ sub: two }), { + ...START, + uploadKey: KEY, + }); + + expect(second.status).toBe(200); + expect(second.body.recordingId).not.toBe(first.body.recordingId); + expect( + Object.fromEntries( + (await rows("recordings")).map((row) => [row.id, row.owner_email]), + ), + ).toEqual({ + [first.body.recordingId]: one, + [second.body.recordingId]: two, + }); + }); + + it("answers a start that raced another with the same key with the upload the other made", async () => { + const editor = await clipsEditor(); + let raced: Awaited> | undefined; + state.duringStorageCheck = async () => { + raced = await startUpload(await token({ sub: editor }), { + ...START, + uploadKey: KEY, + }); + }; + const result = await startUpload(await token({ sub: editor }), { + ...START, + uploadKey: KEY, + }); + + expect(raced?.status).toBe(200); + expect(result.status).toBe(200); + expect(result.body.recordingId).toBe(raced?.body.recordingId); + expect(state.created).toHaveLength(1); + expect(await rows("recordings")).toHaveLength(1); + expect(await rows("recording_content_bindings")).toHaveLength(1); + }); +}); + +describe("a page's final chunk and its cancel", () => { + const KEY = "picked-file-finishing"; + + beforeEach(() => { + // A free clip, so a cancel that fails the upload releases it. + state.rowStorageMode = "hosted"; + state.realAbort = true; + }); + + async function started() { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const answer = await startUpload(bearer, { ...START, uploadKey: KEY }); + expect(answer.status).toBe(200); + const body = answer.body as { + recordingId: string; + chunkUrl: string; + cancelUrl: string; + }; + return { editor, bearer, ...body }; + } + + function finalChunk(upload: { chunkUrl: string }) { + return postChunk(`${upload.chunkUrl}&index=0&isFinal=1`, { + origin: pairedOrigin(), + }); + } + + function cancel(upload: { cancelUrl: string }) { + return postCancel(upload.cancelUrl, pairedOrigin()); + } + + /** What a final chunk in flight holds, without the chunk itself. */ + async function holdClaim( + upload: { editor: string; recordingId: string }, + claimedAt = new Date().toISOString(), + ) { + await asEditor(upload.editor, async () => { + await writeAppState( + `recording-upload-final-chunk-${upload.recordingId}`, + { + claimId: "claim-held", + claimedAt, + }, + ); + await writeAppState(`recording-upload-${upload.recordingId}`, { + recordingId: upload.recordingId, + finalChunkClaimedAt: claimedAt, + }); + }); + } + + it("asks a final chunk that lands while Clips takes an earlier send of it to try again, and takes its bytes once", async () => { + const upload = await started(); + state.finalChunkOutcome = "finish"; + let during: Awaited> | undefined; + state.duringFinalChunk = async () => { + during = await finalChunk(upload); + }; + + const first = await finalChunk(upload); + + expect(first.status).toBe(200); + expect(during).toMatchObject({ + status: 409, + body: { outcome: "upload_finishing", recordingId: upload.recordingId }, + }); + expect(state.chunks).toHaveLength(1); + }); + + it.each([ + ["finished", "finish", 200, { finalized: true, status: "ready" }], + [ + "parked", + "park", + 202, + { finalized: false, waitingForStorage: true, status: "waiting_storage" }, + ], + ] as const)( + "answers a final chunk sent again once the first %s the upload with the first one's success, without taking it again", + async (_phase, outcome, httpStatus, answer) => { + const upload = await started(); + state.finalChunkOutcome = outcome; + const first = await finalChunk(upload); + expect(first).toMatchObject({ status: httpStatus, body: answer }); + + const again = await finalChunk(upload); + + expect(again).toMatchObject({ + status: httpStatus, + body: { ok: true, ...answer }, + }); + expect(state.chunks).toHaveLength(1); + }, + ); + + it("lets a final chunk take over a claim older than a final chunk's longest run", async () => { + const upload = await started(); + await holdClaim(upload, new Date(Date.now() - 10 * 60_000).toISOString()); + state.finalChunkOutcome = "finish"; + + const answer = await finalChunk(upload); + + expect(answer).toMatchObject({ status: 200, body: { ok: true } }); + expect(state.chunks).toHaveLength(1); + }); + + /** An upload whose lease ran out a moment ago, as the reaper finds it. */ + async function expireLease(recordingId: string) { + await client.query( + "UPDATE recordings SET upload_lease_expires_at = $1 WHERE id = $2", + [new Date(Date.now() - 1_000).toISOString(), recordingId], + ); + } + + async function reap() { + // The reaper reads application state before anything here writes it. + await client.exec(`CREATE TABLE IF NOT EXISTS application_state ( + session_id TEXT NOT NULL, + key TEXT NOT NULL, + value TEXT NOT NULL, + updated_at BIGINT NOT NULL, + PRIMARY KEY (session_id, key) + )`); + return withDbExec(coreDb(), () => reapExpiredUploads()); + } + + it("leaves the recording uploading when the reaper runs between a final chunk's claim and Clips taking its bytes, and the chunk finishes", async () => { + state.rowStorageMode = null; + const upload = await started(); + await expireLease(upload.recordingId); + state.finalChunkOutcome = "finish"; + let chunk: ReturnType | undefined; + let claimed!: () => void; + const claimTaken = new Promise((resolve) => { + claimed = resolve; + }); + let reaping: ReturnType | undefined; + // The reaper has read the expired row; the final chunk claims it, and + // Clips takes the bytes only once the reaper is done. + state.beforeReaperFails = async () => { + chunk = finalChunk(upload); + await claimTaken; + }; + state.duringFinalChunk = async () => { + claimed(); + await reaping; + }; + + reaping = reap(); + const reaped = await reaping; + const answer = await chunk; + + expect(reaped.failed).toBe(0); + expect(answer).toMatchObject({ status: 200, body: { ok: true } }); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ id: upload.recordingId, status: "ready" }), + ]); + }); + + it("refuses a final chunk once the reaper failed its expired upload, and changes nothing", async () => { + state.rowStorageMode = null; + const upload = await started(); + await expireLease(upload.recordingId); + expect((await reap()).failed).toBe(1); + const reaped = await rows("recordings"); + expect(reaped).toEqual([ + expect.objectContaining({ + id: upload.recordingId, + status: "failed", + failure_code: "upload_timed_out", + }), + ]); + state.finalChunkOutcome = "finish"; + + const answer = await finalChunk(upload); + + expect(answer.status).toBe(409); + expect(state.chunks).toHaveLength(0); + expect(await rows("recordings")).toEqual(reaped); + }); + + it("asks a cancel that lands while Clips takes the final chunk to try again, changes nothing, and says it finished once the chunk did", async () => { + const upload = await started(); + let during: Awaited> | undefined; + state.duringFinalChunk = async () => { + during = await cancel(upload); + }; + state.finalChunkOutcome = "finish"; + + const chunk = await finalChunk(upload); + + expect(chunk.status).toBe(200); + expect(during).toMatchObject({ + status: 409, + body: { outcome: "upload_finishing", recordingId: upload.recordingId }, + }); + expect(await cancel(upload)).toMatchObject({ + status: 409, + body: { + outcome: "already_finished", + recordingId: upload.recordingId, + status: "ready", + }, + }); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ id: upload.recordingId, status: "ready" }), + ]); + expect(state.released).toEqual([]); + }); + + it("cancels on a cancel's retry once a final chunk that held the upload fails, releasing its free clip", async () => { + const upload = await started(); + let during: Awaited> | undefined; + state.duringFinalChunk = async () => { + during = await cancel(upload); + }; + state.finalChunkOutcome = "fail"; + + const chunk = await finalChunk(upload); + + expect(chunk.status).toBe(502); + expect(during).toMatchObject({ + status: 409, + body: { outcome: "upload_finishing" }, + }); + // The cancel during the chunk changed nothing. + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ status: "uploading" }), + ]); + expect(state.released).toEqual([]); + + expect(await cancel(upload)).toMatchObject({ + status: 200, + body: { outcome: "cancelled", recordingId: upload.recordingId }, + }); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ + status: "failed", + failure_code: "upload_aborted", + }), + ]); + expect(state.released).toEqual([upload.recordingId]); + }); + + it("issues no ticket to a repeat once the final chunk parked the upload until storage is connected", async () => { + const upload = await started(); + state.finalChunkOutcome = "park"; + expect((await finalChunk(upload)).status).toBe(202); + const ticketsBefore = await rows("settings"); + + const repeat = await startUpload(upload.bearer, { + ...START, + uploadKey: KEY, + }); + + expect(repeat).toMatchObject({ + status: 409, + body: { code: "upload_key_used" }, + }); + expect(await rows("settings")).toEqual(ticketsBefore); + }); + + it("cancels an upload its final chunk parked until storage is connected, releasing its free clip", async () => { + const upload = await started(); + let during: Awaited> | undefined; + state.duringFinalChunk = async () => { + during = await cancel(upload); + }; + state.finalChunkOutcome = "park"; + await finalChunk(upload); + + expect(during).toMatchObject({ + status: 409, + body: { outcome: "upload_finishing" }, + }); + expect(await cancel(upload)).toMatchObject({ + status: 200, + body: { outcome: "cancelled", recordingId: upload.recordingId }, + }); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ status: "failed" }), + ]); + expect(state.released).toEqual([upload.recordingId]); + }); + + it("never reads an upload whose final chunk finishes between a repeat's reads as uploading", async () => { + const upload = await started(); + await holdClaim(upload); + // The final chunk finishes, records it and gives the claim up just + // before the repeat reads the claim. + state.beforeClaimRead = async () => { + await client.query("UPDATE recordings SET status = 'ready'"); + await asEditor(upload.editor, () => + writeAppState(`recording-upload-${upload.recordingId}`, { + recordingId: upload.recordingId, + finalChunkOutcome: "finished", + }), + ); + await client.query("DELETE FROM application_state WHERE key = $1", [ + `recording-upload-final-chunk-${upload.recordingId}`, + ]); + }; + const ticketsBefore = await rows("settings"); + + const repeat = await startUpload(upload.bearer, { + ...START, + uploadKey: KEY, + }); + + expect(repeat).toMatchObject({ + status: 409, + body: { + code: "upload_already_finished", + recordingId: upload.recordingId, + }, + }); + expect(await rows("settings")).toEqual(ticketsBefore); + }); + + it("lets a cancel and a repeat take over a claim older than a final chunk's longest run", async () => { + const upload = await started(); + await holdClaim(upload, new Date(Date.now() - 10 * 60_000).toISOString()); + + const repeat = await startUpload(upload.bearer, { + ...START, + uploadKey: KEY, + }); + expect(repeat.status).toBe(200); + expect(await cancel(upload)).toMatchObject({ + status: 200, + body: { outcome: "cancelled" }, + }); + expect(state.released).toEqual([upload.recordingId]); + }); + + it.each([ + [ + "finishes", + "finish" as const, + (recordingId: string) => ({ + status: 409, + body: { + code: "upload_already_finished", + recordingId, + status: "ready", + }, + }), + ], + [ + "fails", + "fail" as const, + (recordingId: string) => ({ status: 200, body: { recordingId } }), + ], + ])( + "asks a repeat that lands while Clips takes the final chunk to try again, issuing no ticket, and answers the next once the chunk %s", + async (_, outcome, next) => { + const upload = await started(); + let repeat: Awaited> | undefined; + let ticketsBefore: unknown; + state.duringFinalChunk = async () => { + ticketsBefore = await rows("settings"); + repeat = await startUpload(upload.bearer, { ...START, uploadKey: KEY }); + }; + state.finalChunkOutcome = outcome; + + await finalChunk(upload); + + expect(repeat).toMatchObject({ + status: 409, + body: { code: "upload_starting" }, + }); + expect(await rows("settings")).toEqual(ticketsBefore); + expect( + await startUpload(upload.bearer, { ...START, uploadKey: KEY }), + ).toMatchObject(next(upload.recordingId)); + }, + ); + + it("refuses a final chunk once a cancel has claimed the upload", async () => { + const upload = await started(); + // A cancel between claiming the upload state and failing the recording. + await asEditor(upload.editor, () => + writeAppState(`recording-upload-${upload.recordingId}`, { + recordingId: upload.recordingId, + status: "failed", + aborted: true, + }), + ); + state.finalChunkOutcome = "finish"; + + const chunk = await finalChunk(upload); + + expect(chunk.status).toBe(409); + expect(state.chunks).toEqual([]); + expect(await rows("recordings")).toEqual([ + expect.objectContaining({ status: "uploading" }), + ]); + }); + + it("puts the upload back to uploading when its final chunk fails, so a repeat and a cancel work again", async () => { + const upload = await started(); + state.finalChunkOutcome = "fail"; + await finalChunk(upload); + + const repeat = await startUpload(upload.bearer, { + ...START, + uploadKey: KEY, + }); + expect(repeat.status).toBe(200); + expect(await cancel(repeat.body)).toMatchObject({ + status: 200, + body: { outcome: "cancelled" }, + }); + expect(state.released).toEqual([upload.recordingId]); + }); +}); + +describe("the storage state Content shows", () => { + const base = { + configured: false, + activeProvider: null, + localSqlChunkScratchAvailable: false, + hostedRemaining: 0, + hostedLimit: 3, + maxClipBytes: 100, + maxClipDurationMs: 1, + builderConfigured: false, + builderUploadConfigured: false, + builderReauthorizationRequired: false, + }; + + it.each([ + [ + "Builder.io storage", + { mode: "own", activeProvider: { id: "builder", name: "Builder.io" } }, + { state: "ready", provider: "builder" }, + ], + [ + "custom keys", + { mode: "own", activeProvider: { id: "s3", name: "S3" } }, + { state: "ready", provider: "custom" }, + ], + [ + "a local Clips", + { + mode: "none", + denyReason: "disabled", + localSqlChunkScratchAvailable: true, + }, + { state: "ready", provider: "local" }, + ], + [ + "free clips left", + { mode: "hosted", hostedRemaining: 2 }, + { state: "free", remaining: 2, limit: 3, maxClipBytes: 100 }, + ], + [ + "free clips used up", + { mode: "none", denyReason: "exhausted" }, + { state: "used_up", limit: 3 }, + ], + [ + "no free storage offered", + { mode: "none", denyReason: "disabled" }, + { state: "none", reason: "disabled" }, + ], + [ + "free storage refused for an unverified email", + { mode: "none", denyReason: "unverified" }, + { state: "none", reason: "unverified" }, + ], + ])("maps %s to its own state", (_label, status, expected) => { + expect( + contentClipStorageFor({ + ...base, + ...status, + } as Parameters[0]), + ).toEqual(expected); + }); +}); + +describe("chunks posted from the editor's browser", () => { + async function upload() { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + return `${body.chunkUrl}&index=0&isFinal=0`; + } + + it("takes a chunk from the paired Content, allowing that origin alone and no credentials", async () => { + const paired = pairedOrigin(); + expect(paired).toMatch(/^https?:\/\//); + const result = await postChunk(await upload(), { + origin: paired, + guardHeaders: { + "access-control-allow-origin": paired, + "access-control-allow-credentials": "true", + }, + }); + + expect(result.status).toBe(200); + expect(state.chunks).toEqual([ + { + owner: expect.objectContaining({ + ownerEmail: expect.any(String), + fromContentPage: true, + }), + }, + ]); + expect(result.headers.get("access-control-allow-origin")).toBe(paired); + expect(result.headers.get("access-control-allow-credentials")).toBeNull(); + expect(result.headers.get("vary")).toBe("Origin"); + expect(result.headers.get("cache-control")).toBe("private, no-store"); + }); + + it("refuses a chunk from any other origin, without allowing it to read the answer", async () => { + const url = await upload(); + const result = await postChunk(url, { + origin: "https://attacker.example.test", + guardHeaders: { + "access-control-allow-origin": "https://attacker.example.test", + "access-control-allow-credentials": "true", + }, + }); + + expect(result.status).toBe(403); + expect(result.headers.get("access-control-allow-origin")).toBeNull(); + expect(result.headers.get("access-control-allow-credentials")).toBeNull(); + expect(state.chunks).toEqual([]); + }); + + it("answers a preflight itself, for the paired Content alone and with no credentials", async () => { + async function preflight(origin: string) { + const event = mockEvent( + `${CLIPS_URL}/api/uploads/rec_1/content-chunk?ticket=t`, + { + method: "OPTIONS", + headers: { origin, "access-control-request-method": "POST" }, + }, + ); + return (await authMiddleware(event)) as Response; + } + const paired = pairedOrigin(); + const allowed = await preflight(paired); + expect(allowed.status).toBe(204); + expect(allowed.headers.get("access-control-allow-origin")).toBe(paired); + expect(allowed.headers.get("access-control-allow-methods")).toBe("POST"); + expect(allowed.headers.get("access-control-allow-credentials")).toBeNull(); + + const other = await preflight("https://other.example.test"); + expect(other.status).toBe(403); + expect(other.headers.get("access-control-allow-origin")).toBeNull(); + expect(other.headers.get("access-control-allow-credentials")).toBeNull(); + + const cancel = (await authMiddleware( + mockEvent(`${CLIPS_URL}/api/uploads/rec_1/content-cancel?ticket=t`, { + method: "OPTIONS", + headers: { origin: paired, "access-control-request-method": "POST" }, + }), + )) as Response; + expect(cancel.status).toBe(204); + expect(cancel.headers.get("access-control-allow-origin")).toBe(paired); + expect(cancel.headers.get("access-control-allow-credentials")).toBeNull(); + }); + + it("answers a bad ticket so the paired page can read why", async () => { + const paired = pairedOrigin(); + const url = new URL(await upload()); + url.searchParams.set("ticket", "clipsup_bm90LWEtdGlja2V0.forged"); + const result = await postChunk(url.href, { origin: paired }); + + expect(result.status).toBe(401); + expect(result.headers.get("access-control-allow-origin")).toBe(paired); + expect(state.chunks).toEqual([]); + }); + + it("refuses a ticket for an upload that didn't come from a Content page", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + await client.query("DELETE FROM recording_content_bindings"); + const result = await postChunk(`${body.chunkUrl}&index=0&isFinal=0`, { + origin: pairedOrigin(), + }); + expect(result.status).toBe(401); + }); + + it("refuses a chunk once the upload's page is in Trash", async () => { + const url = await upload(); + await client.query( + "UPDATE recording_content_bindings SET state = 'trashed'", + ); + const result = await postChunk(url, { origin: pairedOrigin() }); + + expect(result.status).toBe(409); + expect(result.body).toEqual({ error: "This upload's page is in Trash" }); + expect(state.chunks).toEqual([]); + }); + + it("refuses an agent upload's ticket on the Content route", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + await state.db.insert(schema.recordings).values({ + id: "rec-agent", + organizationId: createdOrgId(), + ownerEmail: editor, + title: "An agent's upload", + status: "uploading", + }); + const agentTicket = await withDbExec(coreDb(), () => + createRecordingUploadTicket(editor, "rec-agent"), + ); + const url = new URL(`${body.chunkUrl}&index=0&isFinal=0`); + url.pathname = "/api/uploads/rec-agent/content-chunk"; + url.searchParams.set("ticket", agentTicket.token); + expect((await postChunk(url.href, { origin: pairedOrigin() })).status).toBe( + 401, + ); + }); + + it("refuses a chunk off Content's own route for an upload from a page, and not for a Clip made in Clips", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor })); + + await expect( + refuseContentUploadChunk(body.recordingId), + ).rejects.toMatchObject({ status: 409 }); + await expect( + refuseContentUploadChunk("rec-made-in-clips"), + ).resolves.toBeUndefined(); + }); +}); + +describe("an upload the editor's page gives up", () => { + async function started(blockId = "block-video-1") { + const editor = await clipsEditor(); + const { body } = await startUpload(await token({ sub: editor }), { + ...START, + blockId, + }); + return { editor, ...body } as { + editor: string; + recordingId: string; + cancelUrl: string; + }; + } + + it("fails the upload through Clips' own abort, with its ticket alone", async () => { + const upload = await started(); + const paired = pairedOrigin(); + const result = await postCancel(upload.cancelUrl, paired); + + expect(result.status).toBe(200); + expect(result.body).toEqual({ + outcome: "cancelled", + recordingId: upload.recordingId, + }); + expect(state.aborts).toEqual([ + expect.objectContaining({ + recordingId: upload.recordingId, + ownerEmail: upload.editor, + orgId: expect.stringMatching(/^org-/), + authUserId: expect.stringMatching(/^user-/), + onlyWhileUploading: true, + body: expect.objectContaining({ failureCode: "upload_aborted" }), + }), + ]); + expect((await rows("recordings"))[0]).toMatchObject({ status: "failed" }); + expect(result.headers.get("access-control-allow-origin")).toBe(paired); + expect(result.headers.get("access-control-allow-credentials")).toBeNull(); + }); + + it("answers a repeated cancel the same way, without aborting again", async () => { + const upload = await started(); + const paired = pairedOrigin(); + const first = await postCancel(upload.cancelUrl, paired); + const again = await postCancel(upload.cancelUrl, paired); + + expect(again.status).toBe(200); + expect(again.body).toEqual(first.body); + expect(state.aborts).toHaveLength(1); + }); + + it.each([ + [ + "answers ok while it verifies the file", + { + status: "processing" as const, + answer: { ok: true, verificationPending: true, chunksCleared: 0 }, + }, + ], + [ + "answers ok because the Clip is ready", + { + status: "ready" as const, + answer: { ok: true, alreadyReady: true, chunksCleared: 0 }, + }, + ], + [ + "refuses the abort as stale", + { + status: "processing" as const, + answer: { error: "A newer upload retry is already active." }, + httpStatus: 409, + }, + ], + ])( + "says the upload already finished when Clips finalized it first and the abort %s", + async (_how, finalizedFirst) => { + const upload = await started(); + state.finalizedFirst = finalizedFirst; + const result = await postCancel(upload.cancelUrl, pairedOrigin()); + + expect(result.status).toBe(409); + expect(result.body).toEqual({ + error: "This upload is already finished", + outcome: "already_finished", + recordingId: upload.recordingId, + status: finalizedFirst.status, + }); + expect(state.aborts).toHaveLength(1); + expect((await rows("recordings"))[0]).toMatchObject({ + status: finalizedFirst.status, + }); + }, + ); + + it("asks Content to try again while a final chunk holds the upload, and changes nothing", async () => { + const upload = await started(); + const claimedAt = new Date().toISOString(); + await asEditor(upload.editor, () => + writeAppState(`recording-upload-final-chunk-${upload.recordingId}`, { + claimId: "claim-held", + claimedAt, + }), + ); + const result = await postCancel(upload.cancelUrl, pairedOrigin()); + + expect(result.status).toBe(409); + expect(result.body).toEqual({ + error: "This upload is finishing. Try again.", + outcome: "upload_finishing", + recordingId: upload.recordingId, + }); + expect(state.aborts).toEqual([]); + expect((await rows("recordings"))[0]).toMatchObject({ + status: "uploading", + }); + }); + + it("refuses another recording's ticket and an expired one", async () => { + const one = await started(); + const other = ( + await startUpload(await token({ sub: one.editor }), { + ...START, + blockId: "block-video-2", + }) + ).body as { recordingId: string; cancelUrl: string }; + const paired = pairedOrigin(); + + const swapped = new URL(other.cancelUrl); + swapped.searchParams.set( + "ticket", + required(new URL(one.cancelUrl).searchParams.get("ticket"), "a ticket"), + ); + expect((await postCancel(swapped.href, paired)).status).toBe(401); + + const [setting] = await rows("settings"); + const tickets = JSON.parse(setting.value); + tickets.tickets[one.recordingId].expiresAt = Date.now() - 1; + await client.query("UPDATE settings SET value = $1 WHERE key = $2", [ + JSON.stringify(tickets), + setting.key, + ]); + expect((await postCancel(one.cancelUrl, paired)).status).toBe(401); + + expect(state.aborts).toEqual([]); + expect( + (await rows("recordings")).map((recording) => recording.status), + ).toEqual(["uploading", "uploading"]); + }); + + it("refuses a cancel from any other origin", async () => { + const upload = await started(); + const result = await postCancel( + upload.cancelUrl, + "https://attacker.example.test", + ); + expect(result.status).toBe(403); + expect(result.headers.get("access-control-allow-origin")).toBeNull(); + expect(state.aborts).toEqual([]); + }); + + it("leaves a Clip alone once Clips has its whole file", async () => { + const upload = await started(); + await client.query("UPDATE recordings SET status = 'processing'"); + const result = await postCancel(upload.cancelUrl, pairedOrigin()); + + expect(result.status).toBe(409); + expect(result.body).toMatchObject({ + outcome: "already_finished", + recordingId: upload.recordingId, + status: "processing", + }); + expect(state.aborts).toEqual([]); + }); + + it("still cancels an upload whose page is in Trash", async () => { + const upload = await started(); + await client.query( + "UPDATE recording_content_bindings SET state = 'trashed'", + ); + const result = await postCancel(upload.cancelUrl, pairedOrigin()); + + expect(result.status).toBe(200); + expect(result.body).toMatchObject({ outcome: "cancelled" }); + }); +}); + +describe("From Content and Library in Clips", () => { + async function list( + editor: string, + orgId: string, + view: "library" | "from-content" | "space", + countOnly = false, + where: { folderId?: string; spaceId?: string } = {}, + ) { + return withDbExec(coreDb(), () => + runWithRequestContext({ userEmail: editor, orgId }, () => + listRecordings.run( + { + view, + kind: "all", + sort: "recent", + limit: 100, + offset: 0, + countOnly, + includeMedia: false, + ...where, + }, + { caller: "frontend", userEmail: editor }, + ), + ), + ) as Promise; + } + + it("lists a page's upload under From Content with its page link, and keeps it out of Library", async () => { + const editor = await clipsEditor(); + const { body } = await startUpload( + await token({ sub: editor, documentId: "doc/1" }), + ); + const orgId = createdOrgId(); + for (const id of ["rec-made-in-clips", "rec-linked-into-a-page"]) { + await state.db.insert(schema.recordings).values({ + id, + organizationId: orgId, + ownerEmail: editor, + title: id, + status: "ready", + }); + } + await state.db.insert(schema.recordingContentBindings).values({ + id: "binding-link", + recordingId: "rec-linked-into-a-page", + contentApp: CONTENT_APP, + documentId: "doc-2", + origin: "link", + createdBy: editor, + }); + + const fromContent = await list(editor, orgId, "from-content"); + expect( + fromContent.recordings.map((r: RouteBody) => [r.id, r.contentPageUrl]), + ).toEqual([[body.recordingId, `${CONTENT_APP}/page/doc%2F1`]]); + + const library = await list(editor, orgId, "library"); + expect(library.recordings.map((r: RouteBody) => r.id).sort()).toEqual([ + "rec-linked-into-a-page", + "rec-made-in-clips", + ]); + expect(library.recordings[0]).not.toHaveProperty("contentPageUrl"); + + expect((await list(editor, orgId, "library", true)).total).toBe(2); + expect((await list(editor, orgId, "from-content", true)).total).toBe(1); + }); + + it("lists an upload someone moved into a folder in that folder, and still not in Library", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const moved = (await startUpload(bearer)).body.recordingId; + const kept = ( + await startUpload(bearer, { ...START, blockId: "block-video-2" }) + ).body.recordingId; + const orgId = createdOrgId(); + await client.query( + "UPDATE recordings SET folder_id = 'folder-1' WHERE id = $1", + [moved], + ); + + const folder = await list(editor, orgId, "library", false, { + folderId: "folder-1", + }); + expect(folder.recordings.map((r: RouteBody) => r.id)).toEqual([moved]); + expect( + (await list(editor, orgId, "library", true, { folderId: "folder-1" })) + .total, + ).toBe(1); + expect((await list(editor, orgId, "library")).recordings).toEqual([]); + const fromContent = await list(editor, orgId, "from-content"); + expect(fromContent.recordings.map((r: RouteBody) => r.id).sort()).toEqual( + [moved, kept].sort(), + ); + }); + + it("lists an upload whose page is in Trash in neither From Content nor Library", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const trashed = (await startUpload(bearer)).body.recordingId; + const kept = ( + await startUpload(bearer, { ...START, blockId: "block-video-2" }) + ).body.recordingId; + const orgId = createdOrgId(); + await client.query( + "UPDATE recording_content_bindings SET state = 'trashed' WHERE recording_id = $1", + [trashed], + ); + + const fromContent = await list(editor, orgId, "from-content"); + expect(fromContent.recordings.map((r: RouteBody) => r.id)).toEqual([kept]); + expect((await list(editor, orgId, "from-content", true)).total).toBe(1); + expect([...(await contentPageUrls([trashed, kept])).keys()]).toEqual([ + kept, + ]); + expect((await list(editor, orgId, "library")).recordings).toEqual([]); + expect((await list(editor, orgId, "library", true)).total).toBe(0); + }); + + it("drops an upload from the folder and space someone moved it into while its page is in Trash", async () => { + const editor = await clipsEditor(); + const bearer = await token({ sub: editor }); + const trashed = (await startUpload(bearer)).body.recordingId; + const kept = ( + await startUpload(bearer, { ...START, blockId: "block-video-2" }) + ).body.recordingId; + const orgId = createdOrgId(); + await state.db.insert(schema.recordings).values({ + id: "rec-made-in-clips", + organizationId: orgId, + ownerEmail: editor, + title: "rec-made-in-clips", + status: "ready", + folderId: "folder-1", + spaceIds: '["space-1"]', + }); + await client.query( + `UPDATE recordings SET folder_id = 'folder-1', space_ids = '["space-1"]' WHERE id = ANY($1)`, + [[trashed, kept]], + ); + await client.query( + "UPDATE recording_content_bindings SET state = 'trashed' WHERE recording_id = $1", + [trashed], + ); + + const folder: { recordings: { id: string }[] } = await list( + editor, + orgId, + "library", + false, + { folderId: "folder-1" }, + ); + expect(folder.recordings.map((r) => r.id).sort()).toEqual( + [kept, "rec-made-in-clips"].sort(), + ); + expect( + (await list(editor, orgId, "library", true, { folderId: "folder-1" })) + .total, + ).toBe(2); + const space: { recordings: { id: string }[] } = await list( + editor, + orgId, + "space", + false, + { folderId: "folder-1", spaceId: "space-1" }, + ); + expect(space.recordings.map((r) => r.id).sort()).toEqual( + [kept, "rec-made-in-clips"].sort(), + ); + }); + + it("lists only the signed-in person's own uploads", async () => { + const editor = await clipsEditor(); + expect((await startUpload(await token({ sub: editor }))).status).toBe(200); + const orgId = createdOrgId(); + const someoneElse = await clipsEditor(); + + expect((await list(someoneElse, orgId, "from-content")).recordings).toEqual( + [], + ); + }); +}); diff --git a/templates/clips/server/lib/content-clip-uploads.ts b/templates/clips/server/lib/content-clip-uploads.ts new file mode 100644 index 00000000000..abe757ebf8b --- /dev/null +++ b/templates/clips/server/lib/content-clip-uploads.ts @@ -0,0 +1,977 @@ +/** + * Uploads from an Agent-Native Content document. Content vouches, with an + * upload-scoped token, for an editor of one document; Clips makes a private + * recording owned by that editor, binds it to the document as an upload, and + * hands back a ticketed chunk URL the editor's browser posts the file to, and + * a cancel URL it calls when the page gives the upload up. Content's server + * never carries the video. The contract is in + * `shared/content-clip-contract.ts`. + */ +import { isActionContractError } from "@agent-native/core/action"; +import { getDbExec } from "@agent-native/core/db"; +import { resolveOrgIdForEmail } from "@agent-native/core/org"; +import { + getAppConfig, + getAppProductionUrl, + readBodyWithSizeLimit, + runWithRequestContext, + withConfiguredAppBasePath, +} from "@agent-native/core/server"; +import { getBuiltinAgents } from "@agent-native/core/server/agent-discovery"; +import { resolveLaneEndpoint } from "@agent-native/core/shared/environment-lanes"; +import { MAX_UPLOAD_BYTES } from "@shared/upload-limits.js"; +import { and, eq, isNull } from "drizzle-orm"; +import { + getQuery, + getRequestHeader, + getRouterParam, + HTTPError, + type H3Event, +} from "h3"; +import { z } from "zod"; + +import { + CONTENT_CLIP_CANCEL_OUTCOME, + CONTENT_CLIP_CHUNK_FINISHING, + CONTENT_CLIP_CLAIMS, + CONTENT_CLIP_SCOPES, + CONTENT_CLIP_UPLOAD_FILE_NAME_CHARS, + CONTENT_CLIP_UPLOAD_MIME_TYPES, + CONTENT_CLIP_UPLOAD_REFUSED, + type ContentClipCancelAnswer, + type ContentClipStorage, +} from "../../shared/content-clip-contract.js"; +import { getDb, schema } from "../db/index.js"; +import { handleAbortRecordingUpload } from "../routes/api/uploads/[recordingId]/abort.post.js"; +import { handleRecordingChunk } from "../routes/api/uploads/[recordingId]/chunk.post.js"; +import { + PRIVATE_HEADERS, + verifyContentPeerToken, + type VerifiedContentPeer, +} from "./content-clip-peer.js"; +import { isActiveUploadBinding } from "./content-uploads.js"; +import { + claimFinalChunk, + readUploadPhase, + recordFinalChunkOutcome, + releaseFinalChunk, + type UploadStanding, +} from "./final-chunk-claim.js"; +import { + reissueRecordingUpload, + startRecordingUpload, + type RecordingUploadStart, +} from "./recording-upload-start.js"; +import { + RESERVATION_GRACE_MS, + requireUploadTicketOwnerForToken, + verifyRecordingUploadTicket, + type RecordingUploadTicketOwner, +} from "./recording-upload-ticket.js"; +import { nanoid, ownerEmailMatches } from "./recordings.js"; +import { getClipsVideoStorageStatus } from "./video-storage.js"; + +const MAX_START_BODY_BYTES = 4 * 1024; +const MAX_CLIENT_IP_CHARS = 64; +const BLOCK_ID = /^[A-Za-z0-9_-]{1,128}$/; +const UPLOAD_KEY = /^[A-Za-z0-9_-]{16,64}$/; + +interface Uploader { + peer: VerifiedContentPeer; + email: string; + name: string | null; + /** Content's edge address for the uploader, for free storage's network cap. */ + requestHeaders: Headers; +} + +interface UploaderAccount { + userId: string; + orgId: string; +} + +function json(body: unknown, status = 200, headers?: HeadersInit): Response { + return Response.json(body, { + status, + headers: { + ...PRIVATE_HEADERS, + ...Object.fromEntries(new Headers(headers)), + }, + }); +} + +function refused( + status: number, + code: string, + extra: Record = {}, +): Response { + return json({ error: code, code, ...extra }, status); +} + +async function verifyUploader(event: H3Event): Promise { + const peer = await verifyContentPeerToken(event, [ + CONTENT_CLIP_SCOPES.upload, + ]); + if (peer instanceof Response) return peer; + // An upload names no Clip: Clips makes it. + if (CONTENT_CLIP_CLAIMS.recordingId in peer.claims || !peer.viewerEmail) { + return json({ error: "Unauthorized" }, 401); + } + const clientIp = peer.claims[CONTENT_CLIP_CLAIMS.clientIp]; + const requestHeaders = new Headers(); + if ( + typeof clientIp === "string" && + clientIp.length <= MAX_CLIENT_IP_CHARS && + !/[\s\p{Cc}]/u.test(clientIp) + ) { + requestHeaders.set("x-nf-client-connection-ip", clientIp); + } + return { + peer, + email: peer.viewerEmail, + name: peer.viewerName, + requestHeaders, + }; +} + +/** + * The editor's Clips account and workspace, or null when either is missing. + * Clips never makes either one here: the editor signs up in Clips, which + * makes their workspace the first time they open it. + */ +async function readUploaderAccount( + email: string, +): Promise { + const { rows } = await getDbExec().execute({ + sql: 'SELECT id FROM "user" WHERE lower(email) = ? LIMIT 1', + args: [email.toLowerCase()], + }); + const userId = (rows[0] as { id?: unknown } | undefined)?.id; + if (typeof userId !== "string" || !userId) return null; + const orgId = await resolveOrgIdForEmail(email); + return orgId ? { userId, orgId } : null; +} + +async function runAsUploader( + uploader: Uploader, + account: UploaderAccount, + fn: () => Promise, +): Promise { + return await runWithRequestContext( + { + userEmail: uploader.email, + ...(uploader.name === null ? {} : { userName: uploader.name }), + orgId: account.orgId, + authUserId: account.userId, + }, + fn, + ); +} + +type StorageStatus = Awaited>; + +/** What `create-recording` would do with a new upload, as Content shows it. */ +export function contentClipStorageFor( + status: StorageStatus, +): ContentClipStorage { + if (status.mode === "own") { + return { + state: "ready", + provider: status.activeProvider?.id === "builder" ? "builder" : "custom", + }; + } + // A local development Clips buffers chunks in SQL before any free clip. + if (status.localSqlChunkScratchAvailable) { + return { state: "ready", provider: "local" }; + } + if (status.mode === "hosted") { + return { + state: "free", + remaining: status.hostedRemaining, + limit: status.hostedLimit, + maxClipBytes: status.maxClipBytes, + }; + } + if (status.denyReason === "exhausted") { + return { state: "used_up", limit: status.hostedLimit }; + } + return { state: "none", reason: status.denyReason }; +} + +async function readStorage( + uploader: Uploader, + account: UploaderAccount, +): Promise { + return contentClipStorageFor( + await getClipsVideoStorageStatus({ + email: uploader.email, + userId: account.userId, + requestHeaders: uploader.requestHeaders, + }), + ); +} + +export async function handleContentClipUploadStatus( + event: H3Event, +): Promise { + const uploader = await verifyUploader(event); + if (uploader instanceof Response) return uploader; + const account = await readUploaderAccount(uploader.email); + if (!account) return json({ storage: { state: "account_required" } }); + const storage = await runAsUploader(uploader, account, () => + readStorage(uploader, account), + ); + return json({ storage }); +} + +const uploadStartSchema = z + .object({ + fileName: z.string().trim().min(1).max(CONTENT_CLIP_UPLOAD_FILE_NAME_CHARS), + sizeBytes: z.number().int().positive(), + mimeType: z.enum(CONTENT_CLIP_UPLOAD_MIME_TYPES), + blockId: z.string().regex(BLOCK_ID), + /** One per file the editor picks: a repeated start reaches its upload. */ + uploadKey: z.string().regex(UPLOAD_KEY), + }) + .strict(); + +type UploadStart = z.infer; + +/** The file's name without its extension, which Clips titles the Clip with. */ +function titleFromFileName(fileName: string): string { + const title = fileName.replace(/\.[A-Za-z0-9]{1,8}$/, "").trim(); + return (title || fileName).replace(/[\p{Cc}]/gu, " "); +} + +function uploadUrlFor( + recordingId: string, + route: "content-chunk" | "content-cancel", + query: URLSearchParams, +): string { + const base = withConfiguredAppBasePath(getAppProductionUrl()); + return `${base}/api/uploads/${encodeURIComponent(recordingId)}/${route}?${query}`; +} + +const invalidStart = () => json({ error: "Invalid upload request" }, 400); + +async function readUploadStart( + event: H3Event, +): Promise { + let body: unknown; + try { + body = await readBodyWithSizeLimit(event, MAX_START_BODY_BYTES); + } catch (error) { + if ((error as { statusCode?: unknown }).statusCode === 413) { + return json({ error: "Upload request too large" }, 413); + } + if (error instanceof SyntaxError) return invalidStart(); + throw error; + } + const parsed = uploadStartSchema.safeParse(body); + return parsed.success ? parsed.data : invalidStart(); +} + +/** + * The binding as this start left it. A repeat takes over a start that went + * stale, so by the time a slow start fails its binding may be the repeat's. + */ +function asStartLeftIt(bound: BoundStart) { + const bindings = schema.recordingContentBindings; + return and( + eq(bindings.id, bound.id), + eq(bindings.updatedAt, bound.updatedAt), + ); +} + +async function deleteBinding(bound: BoundStart): Promise { + await getDb() + .delete(schema.recordingContentBindings) + .where(asStartLeftIt(bound)) + .catch((error: unknown) => { + console.warn("[content-clip-uploads] binding cleanup failed", { + id: bound.id, + error: error instanceof Error ? error.message : String(error), + }); + }); +} + +/** + * Marks a binding whose start failed without saying it left nothing. If it + * left no recording, the next repeat starts it again under the same recording + * id; a recording it left answers for itself. + */ +async function markStartFailed(bound: BoundStart): Promise { + const now = new Date().toISOString(); + await getDb() + .update(schema.recordingContentBindings) + .set({ uploadStartFailedAt: now, updatedAt: now }) + .where(asStartLeftIt(bound)) + .catch((error: unknown) => { + // Unmarked, the binding is taken over once it is stale instead. + console.warn("[content-clip-uploads] failed start not marked", { + id: bound.id, + error: error instanceof Error ? error.message : String(error), + }); + }); +} + +/** + * An upload start Clips refused for a reason Content shows, or null. Each + * is raised before a recording or a free clip is made. + */ +async function refusedStart( + error: unknown, + readCurrentStorage: () => Promise, +): Promise { + if (error instanceof HTTPError && error.status === 429) { + return refused(429, CONTENT_CLIP_UPLOAD_REFUSED.tooMany); + } + if (!isActionContractError(error)) return null; + if (error.errorCode === "upload_too_large") { + return refused(413, CONTENT_CLIP_UPLOAD_REFUSED.tooLarge, { + maxBytes: error.details?.maxUploadBytes, + }); + } + // Storage went away, or the last free clip went, between the check and + // the reservation. + if (error.statusCode === 402 || error.statusCode === 403) { + return refused(409, CONTENT_CLIP_UPLOAD_REFUSED.storageRequired, { + storage: await readCurrentStorage(), + }); + } + return null; +} + +function startAnswer(upload: RecordingUploadStart): Response { + upload.chunkQuery.set("ticket", upload.ticket); + return json({ + recordingId: upload.recordingId, + chunkUrl: uploadUrlFor( + upload.recordingId, + "content-chunk", + upload.chunkQuery, + ), + cancelUrl: uploadUrlFor( + upload.recordingId, + "content-cancel", + new URLSearchParams({ ticket: upload.ticket }), + ), + chunkSizeBytes: upload.chunkSizeBytes, + totalChunks: upload.totalChunks, + expiresAt: upload.expiresAt, + }); +} + +const keyUsed = () => refused(409, CONTENT_CLIP_UPLOAD_REFUSED.keyUsed); +const uploadStarting = () => + refused(409, CONTENT_CLIP_UPLOAD_REFUSED.uploadStarting); + +/** A start's binding, which its recording is made for. */ +interface BoundStart { + id: string; + recordingId: string; + updatedAt: string; +} + +/** A bound start with no recording, which failed or stalled. */ +interface StalledStart extends BoundStart { + uploadStartFailedAt: string | null; +} + +/** + * The answer to a start Content repeated for a file it already started: that + * upload, never a second recording. A start that failed or stalled before + * its recording comes back for this one to take over. Null when the key is + * new. + */ +async function repeatedStart( + uploader: Uploader, + start: UploadStart, +): Promise { + const bindings = schema.recordingContentBindings; + const [binding = null] = await getDb() + .select({ + id: bindings.id, + recordingId: bindings.recordingId, + blockId: bindings.blockId, + uploadMimeType: bindings.uploadMimeType, + uploadSizeBytes: bindings.uploadSizeBytes, + uploadStartFailedAt: bindings.uploadStartFailedAt, + updatedAt: bindings.updatedAt, + }) + .from(bindings) + .where( + and( + eq(bindings.contentApp, uploader.peer.contentApp), + eq(bindings.documentId, uploader.peer.documentId), + eq(bindings.createdBy, uploader.email), + eq(bindings.uploadKey, start.uploadKey), + ), + ); + if (!binding) return null; + if ( + binding.blockId !== start.blockId || + binding.uploadMimeType !== start.mimeType || + binding.uploadSizeBytes !== start.sizeBytes + ) { + return keyUsed(); + } + const standing = await readUploadPhase(binding.recordingId, uploader.email); + if (!standing) { + // The latest start bound the key and is making its recording, unless it + // failed or stalled. + return binding.uploadStartFailedAt !== null || + Date.now() - Date.parse(binding.updatedAt) >= RESERVATION_GRACE_MS + ? binding + : uploadStarting(); + } + // A claimed final chunk can still fail and leave the upload open, so only a + // finished one may answer that the file is in. + if (standing.phase === "finishing") return uploadStarting(); + if (standing.phase === "finished") { + return refused(409, CONTENT_CLIP_UPLOAD_REFUSED.alreadyFinished, { + recordingId: binding.recordingId, + status: standing.status === "ready" ? "ready" : "processing", + }); + } + // An upload that ended stays ended, a marked start's included: the file + // starts again under a new key. + if (standing.phase !== "uploading") return keyUsed(); + return startAnswer( + await reissueRecordingUpload({ + recordingId: binding.recordingId, + mimeType: binding.uploadMimeType, + sizeBytes: binding.uploadSizeBytes, + }), + ); +} + +export async function handleContentClipUploadStart( + event: H3Event, +): Promise { + const uploader = await verifyUploader(event); + if (uploader instanceof Response) return uploader; + const start = await readUploadStart(event); + if (start instanceof Response) return start; + + const account = await readUploaderAccount(uploader.email); + if (!account) { + return refused(409, CONTENT_CLIP_UPLOAD_REFUSED.storageRequired, { + storage: { state: "account_required" }, + }); + } + + return runAsUploader(uploader, account, async () => { + // Before the storage check: the first start may have taken the last free + // clip, and a start taken over may hold the one it goes on with. + const repeated = await repeatedStart(uploader, start); + if (repeated instanceof Response) return repeated; + const bound = repeated + ? await takeOverStart(repeated) + : await bindNewStart(uploader, account, start); + if (bound instanceof Response) return bound; + + let upload: Awaited>; + try { + upload = await startRecordingUpload( + { + recordingId: bound.recordingId, + title: titleFromFileName(start.fileName), + visibility: "private", + mimeType: start.mimeType, + sizeBytes: start.sizeBytes, + recordingSource: "file_upload", + }, + { + caller: "a2a", + networkProtocol: "a2a", + userEmail: uploader.email, + orgId: account.orgId, + requestHeaders: uploader.requestHeaders, + }, + ); + } catch (error) { + const response = await refusedStart(error, () => + readStorage(uploader, account), + ); + // Any other failure may have left a recording or a free clip under + // this id, so the binding stays for a repeat to start it again. + const leftNothing = + response !== null || + (isActionContractError(error) && + error.details?.cleanupConfirmed === true); + await (leftNothing ? deleteBinding(bound) : markStartFailed(bound)); + if (response) return response; + throw error; + } + + return startAnswer(upload); + }); +} + +/** + * Checks where a new start's file would go, then binds its key before the + * recording is made, so the recording is never without the document that + * may show it. + */ +async function bindNewStart( + uploader: Uploader, + account: UploaderAccount, + start: UploadStart, +): Promise { + const storage = await readStorage(uploader, account); + if (storage.state !== "ready" && storage.state !== "free") { + return refused(409, CONTENT_CLIP_UPLOAD_REFUSED.storageRequired, { + storage, + }); + } + const maxBytes = + storage.state === "free" ? storage.maxClipBytes : MAX_UPLOAD_BYTES; + if (start.sizeBytes > maxBytes) { + return refused(413, CONTENT_CLIP_UPLOAD_REFUSED.tooLarge, { maxBytes }); + } + + const recordingId = nanoid(); + const bindingId = nanoid(16); + const now = new Date().toISOString(); + const bindings = schema.recordingContentBindings; + const inserted = await getDb() + .insert(bindings) + .values({ + id: bindingId, + recordingId, + contentApp: uploader.peer.contentApp, + documentId: uploader.peer.documentId, + blockId: start.blockId, + origin: "upload", + state: "active", + stateChangedAt: now, + createdBy: uploader.email, + uploadKey: start.uploadKey, + uploadMimeType: start.mimeType, + uploadSizeBytes: start.sizeBytes, + createdAt: now, + updatedAt: now, + }) + .onConflictDoNothing({ + target: [ + bindings.contentApp, + bindings.documentId, + bindings.createdBy, + bindings.uploadKey, + ], + }) + .returning({ id: bindings.id }); + if (inserted.length > 0) { + return { id: bindingId, recordingId, updatedAt: now }; + } + // A start with the same key bound first, since the lookup above. + const repeated = await repeatedStart(uploader, start); + if (repeated instanceof Response) return repeated; + return repeated ? await takeOverStart(repeated) : keyUsed(); +} + +/** + * Takes over a start that failed or stalled, for this one to make its + * recording under the same id. Clips reserves a free clip per recording id, + * so a retake never holds a second. Of repeats that read the same binding, + * one takes it; the rest are told it is starting. + */ +async function takeOverStart( + stalled: StalledStart, +): Promise { + const bindings = schema.recordingContentBindings; + const updatedAt = new Date().toISOString(); + const [taken = null] = await getDb() + .update(bindings) + .set({ uploadStartFailedAt: null, updatedAt }) + .where( + and( + eq(bindings.id, stalled.id), + eq(bindings.updatedAt, stalled.updatedAt), + stalled.uploadStartFailedAt === null + ? isNull(bindings.uploadStartFailedAt) + : eq(bindings.uploadStartFailedAt, stalled.uploadStartFailedAt), + ), + ) + .returning({ id: bindings.id }); + return taken + ? { id: stalled.id, recordingId: stalled.recordingId, updatedAt } + : uploadStarting(); +} + +function hostnameOf(url: string | undefined): string | undefined { + return url && URL.canParse(url) ? new URL(url).hostname : undefined; +} + +/** + * The one Content origin whose pages may post a document's upload chunks: + * the deployment's built-in Content, on Clips' own lane. Never a directory + * entry, and never the request's own `Origin`. + */ +export function pairedContentOrigin(): string | null { + const content = getBuiltinAgents("clips").find((app) => app.id === "content"); + if (!content?.url || !URL.canParse(content.url)) return null; + const url = resolveLaneEndpoint( + content.url, + hostnameOf(getAppConfig().app.url), + ); + return URL.canParse(url) ? new URL(url).origin : null; +} + +const CORS_HEADER_NAMES = [ + "access-control-allow-origin", + "access-control-allow-credentials", + "access-control-allow-methods", + "access-control-allow-headers", + "access-control-expose-headers", +]; + +/** The binding to the Content page a recording was uploaded from, if it was. */ +async function uploadBinding( + recordingId: string, +): Promise<{ origin: string; state: string } | null> { + const bindings = schema.recordingContentBindings; + const [binding = null] = await getDb() + .select({ origin: bindings.origin, state: bindings.state }) + .from(bindings) + .where( + and(eq(bindings.recordingId, recordingId), eq(bindings.origin, "upload")), + ) + .limit(1); + return binding; +} + +function firstQueryValue(value: unknown): string | null { + if (Array.isArray(value)) return firstQueryValue(value[0]); + return typeof value === "string" && value ? value : null; +} + +const CONTENT_UPLOAD_PATH = /\/api\/uploads\/[^/]+\/content-(chunk|cancel)$/; + +/** + * The chunk and cancel routes' answer to a preflight. The auth guard would + * answer it with the app-wide CORS policy, which can allow other origins and + * credentials, so the middleware asks here first. Content's requests never + * need one; a preflight gets the same policy they do. + */ +export function answerContentChunkPreflight( + event: H3Event, +): Response | undefined { + if ( + event.req.method !== "OPTIONS" || + !CONTENT_UPLOAD_PATH.test(event.url.pathname) + ) { + return undefined; + } + const origin = getRequestHeader(event, "origin"); + if (!origin || origin !== pairedContentOrigin()) { + return new Response(null, { status: 403, headers: { vary: "Origin" } }); + } + return new Response(null, { + status: 204, + headers: { + vary: "Origin", + "access-control-allow-origin": origin, + "access-control-allow-methods": "POST", + }, + }); +} + +/** + * The CORS answer to a request from the editor's browser: the paired + * Content's origin alone, never credentials, whatever the auth guard set. + */ +function contentUploadCors(event: H3Event): Record | Response { + const responseHeaders = event.res.headers; + for (const name of CORS_HEADER_NAMES) responseHeaders.delete(name); + for (const [name, value] of Object.entries(PRIVATE_HEADERS)) { + responseHeaders.set(name, value); + } + const origin = getRequestHeader(event, "origin"); + const cors: Record = { vary: "Origin" }; + responseHeaders.set("vary", "Origin"); + if (origin) { + if (origin !== pairedContentOrigin()) { + return json({ error: "Origin not allowed" }, 403, cors); + } + cors["access-control-allow-origin"] = origin; + responseHeaders.set("access-control-allow-origin", origin); + } + return cors; +} + +/** + * One chunk of a document's upload, posted by the editor's browser from the + * paired Content. The ticket in the URL is the only authority: the request + * carries no cookie, and the answer never allows credentials. Content sends + * it as a CORS simple request, so it needs no preflight. + */ +export async function handleContentClipChunk(event: H3Event): Promise { + const cors = contentUploadCors(event); + if (cors instanceof Response) return cors; + + const recordingId = getRouterParam(event, "recordingId"); + if (!recordingId) return json({ error: "Not found" }, 404, cors); + const { isFinal } = getQuery(event); + const final = isFinal === "1" || isFinal === "true"; + try { + const owner = await requireUploadTicketOwnerForToken( + recordingId, + firstQueryValue(getQuery(event).ticket), + ); + const binding = await uploadBinding(recordingId); + if (!binding) { + return json({ error: "Invalid or expired upload ticket" }, 401, cors); + } + if (!isActiveUploadBinding(binding)) { + return json({ error: "This upload's page is in Trash" }, 409, cors); + } + if (final) return await takeFinalChunk(event, owner, cors); + return await handleRecordingChunk(event, { + ...owner, + fromContentPage: true, + }); + } catch (error) { + if (!HTTPError.isError(error)) throw error; + // The ticket's own refusal of an upload Clips is making the Clip of. + const status = (error.data as { status?: unknown } | undefined)?.status; + if ( + final && + error.status === 409 && + (status === "processing" || status === "ready") + ) { + return json(finishedChunk(recordingId, status), 200, cors); + } + return json( + { error: error.message, ...(error.data ? { data: error.data } : {}) }, + error.status, + cors, + ); + } +} + +/** What a final chunk sent again after Clips took the first one answers. */ +function finishedChunk(recordingId: string, status: "processing" | "ready") { + return { ok: true, finalized: true, id: recordingId, status }; +} + +/** + * The final chunk, claimed before Clips takes its bytes: a cancel or another + * send of it that comes while the claim holds is asked to try again, and a + * cancel that came before it refuses the chunk. A chunk Clips accepts + * records where the upload went before the claim comes off; one that fails + * leaves it uploading. A final chunk sent again after that answers with the + * first one's success, and Clips takes nothing more. + */ +async function takeFinalChunk( + event: H3Event, + owner: RecordingUploadTicketOwner, + cors: Record, +): Promise { + const asOwner = (run: () => Promise) => + runWithRequestContext( + { + userEmail: owner.ownerEmail, + orgId: owner.orgId, + ...(owner.authUserId === undefined + ? {} + : { authUserId: owner.authUserId }), + }, + run, + ); + const { recordingId } = owner; + const taken = await asOwner(() => + claimFinalChunk(recordingId, owner.ownerEmail), + ); + if (!taken || ("phase" in taken && taken.phase === "ended")) { + return json({ error: "This upload was cancelled" }, 409, cors); + } + if ("phase" in taken) { + if (taken.phase === "finishing") { + return json( + { + error: "This upload is finishing. Try again.", + outcome: CONTENT_CLIP_CHUNK_FINISHING, + recordingId, + }, + 409, + cors, + ); + } + if (taken.phase === "parked") { + return json( + { + ok: true, + finalized: false, + waitingForStorage: true, + status: "waiting_storage", + id: recordingId, + }, + 202, + cors, + ); + } + return json( + finishedChunk( + recordingId, + taken.status === "ready" ? "ready" : "processing", + ), + 200, + cors, + ); + } + const { claim } = taken; + try { + const answer = (await handleRecordingChunk(event, { + ...owner, + fromContentPage: true, + })) as { + ok?: unknown; + waitingForStorage?: unknown; + } | null; + if (answer?.ok === true) { + await asOwner(() => + recordFinalChunkOutcome( + owner.recordingId, + answer.waitingForStorage === true ? "parked" : "finished", + ), + ); + } + return answer; + } finally { + await asOwner(() => releaseFinalChunk(owner.recordingId, claim)); + } +} + +/** + * Gives up a document's upload at once, from the editor's browser, when the + * page can no longer use it: its block is gone, the editor lost edit access, + * or the file stopped sending. Clips' own abort fails the recording and + * releases its free clip, as the lease reaper would an hour later. The ticket + * is the only authority, as for a chunk, and a repeat answers the same. + */ +export async function handleContentClipCancel( + event: H3Event, +): Promise { + const cors = contentUploadCors(event); + if (cors instanceof Response) return cors; + + const recordingId = getRouterParam(event, "recordingId"); + const token = firstQueryValue(getQuery(event).ticket); + const verified = + recordingId && token + ? await verifyRecordingUploadTicket(recordingId, token) + : null; + // A page in Trash can still give its upload up: cancelling only frees it. + if (!recordingId || !verified || !(await uploadBinding(recordingId))) { + return json({ error: "Invalid or expired upload ticket" }, 401, cors); + } + return giveUp( + event, + recordingId, + verified.ownerEmail, + typeof verified.ticket.authUserId === "string" + ? verified.ticket.authUserId + : undefined, + cors, + ); +} + +/** + * Fails the upload through Clips' own abort while it is uploading or parked. + * Its phase, read before and after, is the answer, never the abort's `ok`. + */ +async function giveUp( + event: H3Event, + recordingId: string, + ownerEmail: string, + authUserId: string | undefined, + cors: Record, +): Promise { + const [recording = null] = await getDb() + .select({ + organizationId: schema.recordings.organizationId, + uploadAttemptId: schema.recordings.uploadAttemptId, + uploadGenerationId: schema.recordings.uploadGenerationId, + }) + .from(schema.recordings) + .where( + and( + eq(schema.recordings.id, recordingId), + ownerEmailMatches(schema.recordings.ownerEmail, ownerEmail), + ), + ); + if (!recording) return json({ error: "Not found" }, 404, cors); + const readPhase = () => + runWithRequestContext( + { + userEmail: ownerEmail, + orgId: recording.organizationId, + ...(authUserId === undefined ? {} : { authUserId }), + }, + () => readUploadPhase(recordingId, ownerEmail), + ); + const settled = cancelAnswer(recordingId, await readPhase(), cors); + if (settled) return settled; + + const result = (await handleAbortRecordingUpload(event, { + recordingId, + ownerEmail, + orgId: recording.organizationId, + ...(authUserId === undefined ? {} : { authUserId }), + onlyWhileUploading: true, + body: { + failureCode: "upload_aborted", + reason: "The Content page gave up this upload.", + attemptId: recording.uploadAttemptId ?? undefined, + uploadGenerationId: recording.uploadGenerationId ?? undefined, + }, + })) as Record; + const answer = cancelAnswer(recordingId, await readPhase(), cors); + if (answer) return answer; + if (result.ok === true) { + return json({ error: "Clips didn't cancel this upload" }, 409, cors); + } + return json(result, event.res.status ?? 409, cors); +} + +/** What a cancel did, once the upload is past `uploading` and `parked`. */ +function cancelAnswer( + recordingId: string, + standing: UploadStanding | null, + cors: Record, +): Response | null { + if (!standing) return json({ error: "Not found" }, 404, cors); + const { phase, status } = standing; + if (phase === "ended") { + const cancelled: ContentClipCancelAnswer = { + outcome: CONTENT_CLIP_CANCEL_OUTCOME.cancelled, + recordingId, + }; + return json(cancelled, 200, cors); + } + if (phase === "finishing") { + const finishing: ContentClipCancelAnswer = { + outcome: CONTENT_CLIP_CANCEL_OUTCOME.finishing, + recordingId, + }; + return json( + { error: "This upload is finishing. Try again.", ...finishing }, + 409, + cors, + ); + } + if (phase === "finished") { + const finished: ContentClipCancelAnswer = { + outcome: CONTENT_CLIP_CANCEL_OUTCOME.alreadyFinished, + recordingId, + status: status === "ready" ? "ready" : "processing", + }; + return json( + { error: "This upload is already finished", ...finished }, + 409, + cors, + ); + } + return null; +} diff --git a/templates/clips/server/lib/content-doc-grant.ts b/templates/clips/server/lib/content-doc-grant.ts index 85e3ece00fa..48d424b5885 100644 --- a/templates/clips/server/lib/content-doc-grant.ts +++ b/templates/clips/server/lib/content-doc-grant.ts @@ -2,6 +2,8 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { resolveAccess, type AccessContext } from "@agent-native/core/sharing"; +import { isActiveUploadBinding } from "./content-uploads.js"; + /** * The Content document binding a request was vouched in through, and the * reader Content vouched for (null for a signed-out reader of a public doc). @@ -35,9 +37,7 @@ export async function contentDocReaderRole( ctx: AccessContext, ): Promise<"viewer" | undefined> { const grant = grants.getStore(); - if (!grant || grant.origin !== "upload" || grant.state !== "active") { - return undefined; - } + if (!grant || !isActiveUploadBinding(grant)) return undefined; if (resource.id !== grant.recordingId) return undefined; const userEmail = ctx.userEmail?.trim().toLowerCase() || null; if (userEmail !== grant.viewerEmail) return undefined; diff --git a/templates/clips/server/lib/content-uploads.ts b/templates/clips/server/lib/content-uploads.ts new file mode 100644 index 00000000000..b081cb54a00 --- /dev/null +++ b/templates/clips/server/lib/content-uploads.ts @@ -0,0 +1,118 @@ +import { and, asc, eq, inArray, sql } from "drizzle-orm"; +import { createError } from "h3"; + +import { getDb, schema } from "../db/index.js"; + +const ACTIVE_UPLOAD = { origin: "upload", state: "active" } as const; + +/** + * A page's upload binding still in force: Content hasn't moved the page to + * Trash. Only such a binding lets the page's readers play the Clip, takes the + * upload's chunks, or lists the Clip under From Content with its page link. + */ +export function isActiveUploadBinding(binding: { + origin: string; + state: string; +}): boolean { + return ( + binding.origin === ACTIVE_UPLOAD.origin && + binding.state === ACTIVE_UPLOAD.state + ); +} + +/** `isActiveUploadBinding` as a condition on `recording_content_bindings`. */ +export function activeUploadBinding() { + const bindings = schema.recordingContentBindings; + return and( + eq(bindings.origin, ACTIVE_UPLOAD.origin), + eq(bindings.state, ACTIVE_UPLOAD.state), + ); +} + +/** + * True for a recording uploaded from an Agent-Native Content page, whatever + * its page's state now. Library's own list leaves all of them out; a Clip + * linked into a page stays in Library. + */ +export function uploadedFromContent() { + const bindings = schema.recordingContentBindings; + return sql`EXISTS (SELECT 1 FROM ${bindings} WHERE ${bindings.recordingId} = ${schema.recordings.id} AND ${bindings.origin} = ${ACTIVE_UPLOAD.origin})`; +} + +/** True for an upload whose page binding is still active: From Content's list. */ +export function activeUploadFromContent() { + const bindings = schema.recordingContentBindings; + return sql`EXISTS (SELECT 1 FROM ${bindings} WHERE ${bindings.recordingId} = ${schema.recordings.id} AND ${activeUploadBinding()})`; +} + +/** + * True for an upload whose page is in Trash. Every Clips list and folder + * count leaves it out, a folder or space someone moved it into included, + * until its page is restored; search still finds it, as it does a Clip in + * Trash. + */ +export function uploadFromTrashedPage() { + return sql`(${uploadedFromContent()} AND NOT ${activeUploadFromContent()})`; +} + +/** + * Refuses a chunk for an upload from a Content page on any route but + * Content's own chunk route, which alone checks the page's binding and + * claims the final chunk. The agent route would otherwise take the upload's + * ticket, an ordinary upload ticket, and the editor's own Clips session + * would reach it as the recording's owner. + */ +export async function refuseContentUploadChunk( + recordingId: string, +): Promise { + const bindings = schema.recordingContentBindings; + const uploads = await getDb() + .select({ id: bindings.id }) + .from(bindings) + .where( + and( + eq(bindings.recordingId, recordingId), + eq(bindings.origin, ACTIVE_UPLOAD.origin), + ), + ) + .limit(1); + if (uploads.length > 0) { + throw createError({ + statusCode: 409, + message: + "This upload came from a Content page. Send its chunks to the chunk URL its start returned.", + }); + } +} + +/** + * The Content page each upload came from, by recording id, for active + * bindings only. Clips keeps no page titles, so a page is named by its link + * alone. + */ +export async function contentPageUrls( + recordingIds: string[], +): Promise> { + const urls = new Map(); + if (!recordingIds.length) return urls; + const bindings = schema.recordingContentBindings; + const pages = await getDb() + .select({ + recordingId: bindings.recordingId, + contentApp: bindings.contentApp, + documentId: bindings.documentId, + }) + .from(bindings) + .where( + and(inArray(bindings.recordingId, recordingIds), activeUploadBinding()), + ) + .orderBy(asc(bindings.createdAt)); + for (const page of pages) { + if (urls.has(page.recordingId)) continue; + urls.set( + page.recordingId, + `${page.contentApp}/page/${encodeURIComponent(page.documentId)}`, + ); + } + return urls; +} diff --git a/templates/clips/server/lib/final-chunk-claim.ts b/templates/clips/server/lib/final-chunk-claim.ts new file mode 100644 index 00000000000..8f4c2880a26 --- /dev/null +++ b/templates/clips/server/lib/final-chunk-claim.ts @@ -0,0 +1,276 @@ +/** + * Where a Content page's upload stands, for every path that has to know: a + * repeated start, a cancel and the Content chunk route. It is kept in three + * places, the final chunk's claim, the upload state and the recording row, + * and only `phaseOf` reads them together. + * + * The final chunk claims the upload before Clips takes its bytes, and writes + * where the upload went into the upload state before it gives the claim up. + * A cancel, or another send of the final chunk, changes nothing while the + * claim is live. The claim and the cancel write the upload state with a + * compare-and-swap, so a cancel that read it before a claim fails its own + * and reads again. + * + * The lease reaper reads none of this: the row's lease decides between it + * and a final chunk. Before it claims, the final chunk renews the lease of a + * row still uploading to outlast the claim, and the reaper fails only a row + * whose lease ran out, each in one conditional update. So either the reaper + * failed the row first and the final chunk finds it ended, or the reaper + * leaves it. + */ +import crypto from "node:crypto"; + +import { + compareAndSetAppState, + compareAndSetManyAppState, + readAppState, +} from "@agent-native/core/application-state"; +import { and, eq, sql } from "drizzle-orm"; +import { createError } from "h3"; + +import { getDb, schema } from "../db/index.js"; +import { ownerEmailMatches } from "./recordings.js"; + +const MAX_CAS_ATTEMPTS = 8; + +/** + * How long a claim holds the upload: a little longer than the final chunk's + * request can run, which is Clips' function timeout (75 s in + * `netlify.toml`). A claim older than this was left by a request that was + * cut off, so the upload counts as uploading again, and the next final + * chunk or cancel takes it over. Content's `CLIP_FINISHING_WAIT_MS`, how + * long it keeps sending a final chunk Clips is still taking, follows it. + */ +export const FINAL_CHUNK_CLAIM_TTL_MS = 2 * 60 * 1000; + +/** + * - `uploading`: Clips is still taking chunks. + * - `finishing`: a final chunk holds the upload. + * - `parked`: Clips has the whole file and waits for storage to be connected. + * - `finished`: Clips is making the Clip, or made it. + * - `ended`: cancelled or failed. + */ +export type UploadPhase = + | "uploading" + | "finishing" + | "parked" + | "finished" + | "ended"; + +export interface UploadStanding { + phase: UploadPhase; + /** The recording's own status. */ + status: string; +} + +const uploadStateKey = (recordingId: string) => + `recording-upload-${recordingId}`; +const claimKey = (recordingId: string) => + `recording-upload-final-chunk-${recordingId}`; + +/** Where the final chunk left the upload, in the upload state. */ +const OUTCOME = "finalChunkOutcome"; + +export interface FinalChunkClaim extends Record { + claimId: string; + claimedAt: string; +} + +function liveClaim( + claim: Record | null, + now: number, +): boolean { + if (!claim) return false; + const claimedAt = + typeof claim.claimedAt === "string" ? Date.parse(claim.claimedAt) : NaN; + // A claim with no readable time can't be told from an abandoned one. + return ( + Number.isFinite(claimedAt) && now - claimedAt < FINAL_CHUNK_CLAIM_TTL_MS + ); +} + +function phaseOf( + claim: Record | null, + uploadState: Record | null, + row: { status: string }, + now: number, +): UploadPhase { + if (liveClaim(claim, now)) return "finishing"; + if ( + row.status === "failed" || + uploadState?.aborted === true || + uploadState?.status === "failed" + ) { + return "ended"; + } + if ( + row.status === "processing" || + row.status === "ready" || + uploadState?.[OUTCOME] === "finished" + ) { + return "finished"; + } + return uploadState?.[OUTCOME] === "parked" ? "parked" : "uploading"; +} + +async function readRow(recordingId: string, ownerEmail: string) { + const [row = null] = await getDb() + .select({ status: schema.recordings.status }) + .from(schema.recordings) + .where( + and( + eq(schema.recordings.id, recordingId), + ownerEmailMatches(schema.recordings.ownerEmail, ownerEmail), + ), + ); + return row; +} + +/** + * Where the upload stands, read in its owner's request. Null when the + * recording is gone. The claim is read first, the upload state next and the + * row last: the final chunk writes the row and its outcome before it gives + * the claim up, so one that finishes between two reads shows in the later. + */ +export async function readUploadPhase( + recordingId: string, + ownerEmail: string, + options: { now?: number } = {}, +): Promise { + const claim = await readAppState(claimKey(recordingId)); + const uploadState = await readAppState(uploadStateKey(recordingId)); + const row = await readRow(recordingId, ownerEmail); + if (!row) return null; + return { + phase: phaseOf(claim, uploadState, row, options.now ?? Date.now()), + status: row.status, + }; +} + +/** + * Keeps the row's lease for at least a claim's life, while it is uploading. + * False once something else ended or finished it. + */ +async function holdLease( + recordingId: string, + ownerEmail: string, +): Promise { + const lease = schema.recordings.uploadLeaseExpiresAt; + const until = new Date(Date.now() + FINAL_CHUNK_CLAIM_TTL_MS).toISOString(); + const held = await getDb() + .update(schema.recordings) + .set({ + uploadLeaseExpiresAt: sql`CASE WHEN ${lease} > ${until} THEN ${lease} ELSE ${until} END`, + }) + .where( + and( + eq(schema.recordings.id, recordingId), + eq(schema.recordings.status, "uploading"), + ownerEmailMatches(schema.recordings.ownerEmail, ownerEmail), + ), + ) + .returning({ id: schema.recordings.id }); + return held.length > 0; +} + +function contention(): never { + throw createError({ + statusCode: 503, + message: "This upload is changing. Try again.", + }); +} + +/** + * Claims the upload for its final chunk, before Clips takes the bytes, while + * it is `uploading`; an expired claim counts as none. The row's lease is + * renewed first, so the reaper can't fail it under the claim. Otherwise the + * phase it is in, which the chunk is answered with, and null once it is gone. + */ +export async function claimFinalChunk( + recordingId: string, + ownerEmail: string, +): Promise<{ claim: FinalChunkClaim } | UploadStanding | null> { + const claim: FinalChunkClaim = { + claimId: crypto.randomUUID(), + claimedAt: new Date().toISOString(), + }; + for (let attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt += 1) { + const current = await readAppState(claimKey(recordingId)); + const uploadState = await readAppState(uploadStateKey(recordingId)); + const row = await readRow(recordingId, ownerEmail); + if (!row) return null; + const phase = phaseOf(current, uploadState, row, Date.now()); + if (phase !== "uploading") return { phase, status: row.status }; + // A lease renewed for a claim that then loses its compare-and-swap only + // runs two minutes longer, which nothing needs undone. One that finds no + // row uploading reads again, to answer how it ended. + if (!(await holdLease(recordingId, ownerEmail))) continue; + // The upload state changes too, so a cancel that read it before this + // claim fails its own compare-and-swap and reads the claim. + const claimed = await compareAndSetManyAppState([ + { + key: uploadStateKey(recordingId), + expectedValue: uploadState, + nextValue: { + ...(uploadState ?? { recordingId }), + finalChunkClaimedAt: claim.claimedAt, + }, + }, + { key: claimKey(recordingId), expectedValue: current, nextValue: claim }, + ]); + if (claimed) return { claim }; + } + return contention(); +} + +/** + * Writes where an accepted final chunk left the upload, before its claim + * comes off: `parked` while Clips waits for storage, `finished` once it is + * making the Clip or verifying the file. + */ +export async function recordFinalChunkOutcome( + recordingId: string, + outcome: "parked" | "finished", +): Promise { + for (let attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt += 1) { + const uploadState = await readAppState(uploadStateKey(recordingId)); + if ( + await compareAndSetAppState(uploadStateKey(recordingId), uploadState, { + ...(uploadState ?? { recordingId }), + [OUTCOME]: outcome, + }) + ) { + return; + } + } + return contention(); +} + +/** + * An upload state taking chunks again, as a resumed upload's does, without + * the outcome an earlier final chunk left: that upload isn't finished or + * parked any more, and its phase would otherwise still say so. + */ +export function withoutFinalChunkOutcome( + uploadState: Record, +): Record { + const { [OUTCOME]: _outcome, ...reopened } = uploadState; + return reopened; +} + +/** Gives the claim up once its final chunk is done with. */ +export async function releaseFinalChunk( + recordingId: string, + claim: FinalChunkClaim, +): Promise { + for (let attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt += 1) { + const current = await readAppState(claimKey(recordingId)); + // This claim expired and a later final chunk or cancel took it over; + // that one gives it up. + if (current?.claimId !== claim.claimId) return; + if (await compareAndSetAppState(claimKey(recordingId), current, null)) { + return; + } + } + return contention(); +} diff --git a/templates/clips/server/lib/recording-upload-start.ts b/templates/clips/server/lib/recording-upload-start.ts new file mode 100644 index 00000000000..557ca6f5810 --- /dev/null +++ b/templates/clips/server/lib/recording-upload-start.ts @@ -0,0 +1,141 @@ +import type { ActionRunContext } from "@agent-native/core/action"; +import { getRequestContext } from "@agent-native/core/server"; +import { UPLOAD_SLICE_BYTES } from "@shared/recording-core.js"; +import type { z } from "zod"; + +import createRecording from "../../actions/create-recording.js"; +import type { createRecordingSchema } from "../../actions/lib/create-recording-schema.js"; +import { + addRecordingUploadTicket, + createRecordingUploadTicket, + releaseRecordingUploadTicket, +} from "./recording-upload-ticket.js"; +import { getCurrentOwnerEmail, nanoid } from "./recordings.js"; + +type CreateRecordingArgs = z.input; + +interface UploadFile { + mimeType: string; + sizeBytes: number; + durationMs?: number | undefined; + width?: number | undefined; + height?: number | undefined; +} + +export interface RecordingUploadStart { + recordingId: string; + /** The upload ticket: the only thing a chunk request needs. */ + ticket: string; + expiresAt: string; + chunkSizeBytes: number; + totalChunks: number; + /** The chunk route's query, without `index` and `isFinal`. */ + chunkQuery: URLSearchParams; +} + +/** + * Makes a recording in "uploading" for the signed-in owner and the ticket that + * uploads its file in chunks, the way an agent's upload starts. + */ +export async function startRecordingUpload( + input: { + /** Made up when absent. */ + recordingId?: string; + title?: string | undefined; + visibility: NonNullable; + recordingSource: NonNullable; + } & UploadFile, + actionContext?: ActionRunContext, +): Promise { + const ownerEmail = getCurrentOwnerEmail(); + // The ticket comes first: if it is refused or fails to save, no recording + // is left stranded in "uploading" with nothing able to finish it. + const recordingId = input.recordingId ?? nanoid(); + const ticket = await createRecordingUploadTicket( + ownerEmail, + recordingId, + getRequestContext()?.authUserId, + ); + try { + await createRecording.run( + { + id: recordingId, + title: input.title, + visibility: input.visibility, + mimeType: input.mimeType, + sizeBytes: input.sizeBytes, + width: input.width, + height: input.height, + hasAudio: true, + hasCamera: false, + recordingSource: input.recordingSource, + requestStreaming: true, + }, + actionContext, + ); + } catch (error) { + // create-recording fails its own half-made row, or the lease reaper does; + // only the ticket, which the caller never received, is ours to release. + await releaseRecordingUploadTicket( + ownerEmail, + recordingId, + ticket.tokenHash, + ).catch((releaseError) => { + console.warn("[recording-upload] ticket release failed", { + recordingId, + error: + releaseError instanceof Error + ? releaseError.message + : String(releaseError), + }); + }); + throw error; + } + + return { + recordingId, + ticket: ticket.token, + expiresAt: new Date(ticket.expiresAt).toISOString(), + ...uploadPlan(input), + }; +} + +/** + * Another ticket for an upload already in "uploading", planned for the file + * its first start described. Earlier tickets keep working. + */ +export async function reissueRecordingUpload( + input: { recordingId: string } & UploadFile, +): Promise { + const ticket = await addRecordingUploadTicket( + getCurrentOwnerEmail(), + input.recordingId, + getRequestContext()?.authUserId, + ); + return { + recordingId: input.recordingId, + ticket: ticket.token, + expiresAt: new Date(ticket.expiresAt).toISOString(), + ...uploadPlan(input), + }; +} + +function uploadPlan( + input: UploadFile, +): Pick { + const totalChunks = Math.max( + 1, + Math.ceil(input.sizeBytes / UPLOAD_SLICE_BYTES), + ); + const chunkQuery = new URLSearchParams({ + mimeType: input.mimeType, + total: String(totalChunks), + hasAudio: "1", + hasCamera: "0", + }); + if (input.durationMs) chunkQuery.set("durationMs", String(input.durationMs)); + if (input.width) chunkQuery.set("width", String(input.width)); + if (input.height) chunkQuery.set("height", String(input.height)); + + return { chunkSizeBytes: UPLOAD_SLICE_BYTES, totalChunks, chunkQuery }; +} diff --git a/templates/clips/server/lib/recording-upload-ticket.test.ts b/templates/clips/server/lib/recording-upload-ticket.test.ts index fa921f67760..d8effc1bbd3 100644 --- a/templates/clips/server/lib/recording-upload-ticket.test.ts +++ b/templates/clips/server/lib/recording-upload-ticket.test.ts @@ -345,8 +345,16 @@ describe("recording upload tickets", () => { "rec-2", ); - await releaseRecordingUploadTicket("owner@example.com", "rec-2"); - await releaseRecordingUploadTicket("owner@example.com", "rec-2"); + await releaseRecordingUploadTicket( + "owner@example.com", + "rec-2", + released.tokenHash, + ); + await releaseRecordingUploadTicket( + "owner@example.com", + "rec-2", + released.tokenHash, + ); expect(Object.keys(JSON.parse(settings.get(storageKey)!).tickets)).toEqual([ "rec-1", @@ -359,6 +367,32 @@ describe("recording upload tickets", () => { ).resolves.toMatchObject({ recordingId: "rec-1" }); }); + it("leaves a recording's ticket alone once a later start replaced the one being released", async () => { + const { + createRecordingUploadTicket, + releaseRecordingUploadTicket, + requireRecordingUploadTicketOwner, + } = await import("./recording-upload-ticket.js"); + const stale = await createRecordingUploadTicket( + "owner@example.com", + "rec-1", + ); + const later = await createRecordingUploadTicket( + "owner@example.com", + "rec-1", + ); + + await releaseRecordingUploadTicket( + "owner@example.com", + "rec-1", + stale.tokenHash, + ); + + await expect( + requireRecordingUploadTicketOwner(ticketEvent(later.token), "rec-1"), + ).resolves.toMatchObject({ recordingId: "rec-1" }); + }); + it("keeps other recordings' tickets when issuing a new one", async () => { const { createRecordingUploadTicket, requireRecordingUploadTicketOwner } = await import("./recording-upload-ticket.js"); @@ -372,4 +406,43 @@ describe("recording upload tickets", () => { requireRecordingUploadTicketOwner(ticketEvent(first.token), "rec-1"), ).resolves.toMatchObject({ recordingId: "rec-1" }); }); + + it("replaces a recording's token for a new agent ticket, and keeps it alongside a repeated start's", async () => { + const { + addRecordingUploadTicket, + createRecordingUploadTicket, + requireRecordingUploadTicketOwner, + } = await import("./recording-upload-ticket.js"); + const opens = (token: string) => + requireRecordingUploadTicketOwner(ticketEvent(token), "rec-1").then( + () => true, + () => false, + ); + + const replaced = await createRecordingUploadTicket( + "owner@example.com", + "rec-1", + "user-1", + ); + const agent = await createRecordingUploadTicket( + "owner@example.com", + "rec-1", + "user-1", + ); + const repeat = await addRecordingUploadTicket( + "owner@example.com", + "rec-1", + "user-1", + ); + + expect(await opens(replaced.token)).toBe(false); + expect(await opens(agent.token)).toBe(true); + expect(await opens(repeat.token)).toBe(true); + const stored = JSON.parse(settings.get(storageKey) ?? ""); + expect(Object.keys(stored.tickets)).toEqual(["rec-1"]); + expect(stored.tickets["rec-1"]).toMatchObject({ + reservedAt: agent.reservedAt, + authUserId: "user-1", + }); + }); }); diff --git a/templates/clips/server/lib/recording-upload-ticket.ts b/templates/clips/server/lib/recording-upload-ticket.ts index 2b802690b0f..857ce69e018 100644 --- a/templates/clips/server/lib/recording-upload-ticket.ts +++ b/templates/clips/server/lib/recording-upload-ticket.ts @@ -20,15 +20,26 @@ const MAX_CAS_ATTEMPTS = 8; // How long a reservation whose recording row is still missing counts toward the // cap: ample for create-recording, short enough that a deleted recording or a // failed release does not hold a slot for the ticket's whole lifetime. -const RESERVATION_GRACE_MS = 10 * 60 * 1000; +export const RESERVATION_GRACE_MS = 10 * 60 * 1000; +// The most tokens one recording's ticket keeps live when a start is repeated, +// counting the newest. +const MAX_TOKENS_PER_TICKET = 4; + +interface UploadToken { + tokenHash: string; + expiresAt: number; +} export interface RecordingUploadTicket extends Record { recordingId: string; + /** The newest token's hash; the ticket expires with it. */ tokenHash: string; reservedAt: number; expiresAt: number; /** The creator's auth user id, which a hosted-storage reservation is held under. */ authUserId?: string; + /** Tokens an earlier answer to a repeated start returned, newest first. */ + earlierTokens?: UploadToken[]; } interface RecordingUploadTickets extends Record { @@ -176,29 +187,86 @@ async function compareAndSwapTickets( return result.rowsAffected === 1; } +function liveTokens(ticket: RecordingUploadTicket | undefined): UploadToken[] { + if (!ticket) return []; + const now = Date.now(); + // Parsed from stored JSON, so an earlier token can be anything. + const earlier: unknown[] = Array.isArray(ticket.earlierTokens) + ? ticket.earlierTokens + : []; + return [ + { tokenHash: ticket.tokenHash, expiresAt: ticket.expiresAt }, + ...earlier, + ].filter((token): token is UploadToken => { + const { tokenHash, expiresAt } = (token ?? {}) as Partial; + return ( + typeof tokenHash === "string" && + typeof expiresAt === "number" && + Number.isFinite(expiresAt) && + expiresAt >= now + ); + }); +} + /** * Reserve before creating the recording: the cap is enforced inside the same - * compare-and-swap, so a refusal or a failed write leaves nothing behind. + * compare-and-swap, so a refusal or a failed write leaves nothing behind. The + * new token replaces the recording's earlier ones. + */ +export function createRecordingUploadTicket( + ownerEmail: string, + recordingId: string, + authUserId?: string, +): Promise { + return issueRecordingUploadTicket(ownerEmail, recordingId, authUserId, false); +} + +/** + * Another token for a recording's upload, for a repeated start. The earlier + * tokens keep working, since overlapping repeats each hand one back, up to + * `MAX_TOKENS_PER_TICKET` with the oldest retired first. The recording still + * holds one slot. */ -export async function createRecordingUploadTicket( +export function addRecordingUploadTicket( ownerEmail: string, recordingId: string, authUserId?: string, +): Promise { + return issueRecordingUploadTicket(ownerEmail, recordingId, authUserId, true); +} + +async function issueRecordingUploadTicket( + ownerEmail: string, + recordingId: string, + authUserId: string | undefined, + keepEarlier: boolean, ): Promise { const token = `${TOKEN_PREFIX}${encodeOwner(ownerEmail)}.${crypto.randomBytes(32).toString("base64url")}`; - const reservedAt = Date.now(); - const ticket: RecordingUploadTicket = { - recordingId, - tokenHash: tokenHash(token), - reservedAt, - expiresAt: reservedAt + RECORDING_UPLOAD_TICKET_TTL_MS, - ...(authUserId ? { authUserId } : {}), - }; + const issuedAt = Date.now(); for (let attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt += 1) { const { raw, collection } = await readTicketsRow(ownerEmail); const live = await liveTickets(ownerEmail, collection.tickets); + const earlier = keepEarlier ? live[recordingId] : undefined; delete live[recordingId]; + const ticketAuthUserId = + (typeof earlier?.authUserId === "string" ? earlier.authUserId : "") || + authUserId; + const ticket: RecordingUploadTicket = { + recordingId, + tokenHash: tokenHash(token), + reservedAt: earlier?.reservedAt ?? issuedAt, + expiresAt: issuedAt + RECORDING_UPLOAD_TICKET_TTL_MS, + ...(ticketAuthUserId ? { authUserId: ticketAuthUserId } : {}), + ...(earlier + ? { + earlierTokens: liveTokens(earlier).slice( + 0, + MAX_TOKENS_PER_TICKET - 1, + ), + } + : {}), + }; if (Object.keys(live).length >= MAX_PENDING_TICKETS) { throw createError({ statusCode: 429, @@ -213,14 +281,19 @@ export async function createRecordingUploadTicket( throw new Error("Could not reserve a recording upload ticket"); } -/** For a reservation whose recording was never created. */ +/** + * For a reservation whose recording was never created, while it is still the + * one that token's start made: a later start for the same recording id + * replaces it, and that start's ticket stays. + */ export async function releaseRecordingUploadTicket( ownerEmail: string, recordingId: string, + issuedTokenHash: string, ): Promise { for (let attempt = 0; attempt < MAX_CAS_ATTEMPTS; attempt += 1) { const { raw, collection } = await readTicketsRow(ownerEmail); - if (!collection.tickets[recordingId]) return; + if (collection.tickets[recordingId]?.tokenHash !== issuedTokenHash) return; const { [recordingId]: _released, ...tickets } = collection.tickets; if (await compareAndSwapTickets(ownerEmail, raw, { tickets })) return; } @@ -236,12 +309,10 @@ export async function verifyRecordingUploadTicket( if (!ownerEmail) return null; const { collection } = await readTicketsRow(ownerEmail); const ticket = collection.tickets[recordingId]; + const presented = tokenHash(token); if ( !ticket || - typeof ticket.tokenHash !== "string" || - !hashesMatch(ticket.tokenHash, tokenHash(token)) || - !Number.isFinite(ticket.expiresAt) || - ticket.expiresAt < Date.now() + !liveTokens(ticket).some((live) => hashesMatch(live.tokenHash, presented)) ) { return null; } @@ -259,7 +330,14 @@ export async function requireRecordingUploadTicketOwner( event: H3Event, recordingId: string, ): Promise { - const token = readTicketToken(event); + return requireUploadTicketOwnerForToken(recordingId, readTicketToken(event)); +} + +/** The same check for a ticket that arrives some other way than the header. */ +export async function requireUploadTicketOwnerForToken( + recordingId: string, + token: string | null, +): Promise { const verified = token ? await verifyRecordingUploadTicket(recordingId, token) : null; diff --git a/templates/clips/server/middleware/auth.ts b/templates/clips/server/middleware/auth.ts index f1b0e4eeeec..16eefb8827d 100644 --- a/templates/clips/server/middleware/auth.ts +++ b/templates/clips/server/middleware/auth.ts @@ -1,6 +1,8 @@ import { runAuthGuard } from "@agent-native/core/server"; import { defineEventHandler } from "h3"; +import { answerContentChunkPreflight } from "../lib/content-clip-uploads.js"; + export default defineEventHandler(async (event) => { - return runAuthGuard(event); + return answerContentChunkPreflight(event) ?? runAuthGuard(event); }); diff --git a/templates/clips/server/plugins/auth.ts b/templates/clips/server/plugins/auth.ts index f9bdffef836..e463ba30ad5 100644 --- a/templates/clips/server/plugins/auth.ts +++ b/templates/clips/server/plugins/auth.ts @@ -31,10 +31,17 @@ export default createToolkitAuthPlugin({ // Chunk POSTs from an outside agent carry no session cookie; the route // refuses anything without a ticket for its own recording. "/api/uploads/:recordingId/agent-chunk", + // A document's upload chunks, and its cancel, come from the editor's + // browser with no cookie; each route takes only a ticket for its own + // Content upload. + "/api/uploads/:recordingId/content-chunk", + "/api/uploads/:recordingId/content-cancel", // Content calls these server to server; each refuses anything without a - // Content token naming that recording. + // Content token naming that recording, or, for an upload, that editor. "/api/content-clips/:recordingId/playback", "/api/content-clips/:recordingId/comments", + "/api/content-clips/uploads", + "/api/content-clips/upload-status", "/__manifest", "/api/view-event", "/api/public-recording", diff --git a/templates/clips/server/plugins/db.ts b/templates/clips/server/plugins/db.ts index 4a488098d45..ca0ebc1e069 100644 --- a/templates/clips/server/plugins/db.ts +++ b/templates/clips/server/plugins/db.ts @@ -1371,6 +1371,24 @@ export const migrations = runMigrations( CREATE INDEX IF NOT EXISTS recording_content_bindings_document_idx ON recording_content_bindings (content_app, document_id)`, }, + { + version: 96, + name: "recording-content-binding-upload-key", + sql: `ALTER TABLE recording_content_bindings ADD COLUMN IF NOT EXISTS upload_key TEXT; + CREATE UNIQUE INDEX IF NOT EXISTS recording_content_bindings_upload_key_unique_idx + ON recording_content_bindings (content_app, document_id, created_by, upload_key)`, + }, + { + version: 97, + name: "recording-content-binding-upload-file", + sql: `ALTER TABLE recording_content_bindings ADD COLUMN IF NOT EXISTS upload_mime_type TEXT; + ALTER TABLE recording_content_bindings ADD COLUMN IF NOT EXISTS upload_size_bytes BIGINT`, + }, + { + version: 98, + name: "recording-content-binding-upload-start-failed", + sql: `ALTER TABLE recording_content_bindings ADD COLUMN IF NOT EXISTS upload_start_failed_at TEXT`, + }, ]), { table: "clips_migrations" }, ); diff --git a/templates/clips/server/routes/api/content-clips/upload-status.get.ts b/templates/clips/server/routes/api/content-clips/upload-status.get.ts new file mode 100644 index 00000000000..664fb72166a --- /dev/null +++ b/templates/clips/server/routes/api/content-clips/upload-status.get.ts @@ -0,0 +1,6 @@ +// guard:allow-api-route — Content calls this server to server with a document-scoped A2A token, which no action route accepts. +import { defineEventHandler } from "h3"; + +import { handleContentClipUploadStatus } from "../../../lib/content-clip-uploads.js"; + +export default defineEventHandler(handleContentClipUploadStatus); diff --git a/templates/clips/server/routes/api/content-clips/uploads.post.ts b/templates/clips/server/routes/api/content-clips/uploads.post.ts new file mode 100644 index 00000000000..3c608aa14ac --- /dev/null +++ b/templates/clips/server/routes/api/content-clips/uploads.post.ts @@ -0,0 +1,6 @@ +// guard:allow-api-route — Content calls this server to server with a document-scoped A2A token, which no action route accepts. +import { defineEventHandler } from "h3"; + +import { handleContentClipUploadStart } from "../../../lib/content-clip-uploads.js"; + +export default defineEventHandler(handleContentClipUploadStart); diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.test.ts b/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.test.ts index cfa4edc36fd..6c0e33e942c 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.test.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.test.ts @@ -1,8 +1,10 @@ +import type { H3Event } from "h3"; import { beforeEach, describe, expect, it, vi } from "vitest"; const mockWriteAppState = vi.hoisted(() => vi.fn()); const mockReadAppState = vi.hoisted(() => vi.fn()); const mockCompareAndSetManyAppState = vi.hoisted(() => vi.fn()); +const mockCompareAndSetAppState = vi.hoisted(() => vi.fn()); const mockTrack = vi.hoisted(() => vi.fn()); const mockDeleteRecordingChunks = vi.hoisted(() => vi.fn()); const mockGetRouterParam = vi.hoisted(() => vi.fn()); @@ -50,6 +52,8 @@ const mockDb = vi.hoisted(() => ({ })); vi.mock("@agent-native/core/application-state", () => ({ + compareAndSetAppState: (...args: unknown[]) => + mockCompareAndSetAppState(...args), compareAndSetManyAppState: (...args: unknown[]) => mockCompareAndSetManyAppState(...args), readAppState: (...args: unknown[]) => mockReadAppState(...args), @@ -140,7 +144,7 @@ vi.mock("../../../../lib/resumable-upload-provider.js", () => ({ mockResolveResumableUploadProvider(...args), })); -import handler from "./abort.post"; +import handler, { handleAbortRecordingUpload } from "./abort.post"; describe("/api/uploads/:recordingId/abort route", () => { beforeEach(() => { @@ -167,6 +171,7 @@ describe("/api/uploads/:recordingId/abort route", () => { mockOwnerEmailMatches.mockReturnValue("owner-match"); mockDeleteRecordingChunks.mockResolvedValue(2); mockCompareAndSetManyAppState.mockResolvedValue(true); + mockCompareAndSetAppState.mockResolvedValue(true); mockDeleteResumableSession.mockResolvedValue(undefined); mockGetResumableSession.mockResolvedValue(null); mockAbortSession.mockResolvedValue(undefined); @@ -729,6 +734,222 @@ describe("/api/uploads/:recordingId/abort route", () => { ); }); + it("takes the request from a caller that read it, and releases the hosted clip", async () => { + mockSelectRows.rows = [ + { + id: "rec-1", + status: "uploading", + videoUrl: null, + failureReason: null, + uploadAttemptId: "attempt-1", + uploadGenerationId: "generation-1", + storageMode: "hosted", + }, + ]; + mockReadAppState.mockResolvedValue({ + recordingId: "rec-1", + status: "uploading", + uploadAttemptId: "attempt-1", + uploadGenerationId: "generation-1", + }); + mockGetResumableSession.mockResolvedValue({ + providerId: "clips-hosted", + sessionId: "upload-example", + meta: { objectKey: "clips/rec-1.webm" }, + bytesUploaded: 123, + }); + + await expect( + handleAbortRecordingUpload({} as H3Event, { + recordingId: "rec-1", + ownerEmail: "owner@example.com", + orgId: "org-1", + authUserId: "user-1", + body: { + failureCode: "upload_aborted", + reason: "The Content page gave up this upload.", + attemptId: "attempt-1", + uploadGenerationId: "generation-1", + }, + }), + ).resolves.toMatchObject({ ok: true, recordingId: "rec-1" }); + + expect(mockReadBody).not.toHaveBeenCalled(); + expect(mockGetEventOwnerContext).not.toHaveBeenCalled(); + expect(mockUpdateSets).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: "failed", + failureCode: "upload_aborted", + failureReason: "The Content page gave up this upload.", + }), + ]), + ); + expect(mockReleaseHostedClipRecording).toHaveBeenCalledWith( + expect.objectContaining({ recordingId: "rec-1" }), + ); + }); + + describe("for Content's cancel, which changes only an upload still uploading", () => { + const uploading = { + id: "rec-1", + status: "uploading", + videoUrl: null, + failureReason: null, + failureCode: null, + uploadAttemptId: "attempt-1", + uploadGenerationId: "generation-1", + }; + const uploadState = { + recordingId: "rec-1", + status: "uploading", + uploadAttemptId: "attempt-1", + uploadGenerationId: "generation-1", + }; + // What a final chunk holds from before it takes the bytes until it has + // recorded where the upload went. + const claim = { + claimId: "claim-1", + claimedAt: new Date().toISOString(), + }; + const claimKey = "recording-upload-final-chunk-rec-1"; + + function contentCancel() { + return handleAbortRecordingUpload({} as H3Event, { + recordingId: "rec-1", + ownerEmail: "owner@example.com", + orgId: "org-1", + onlyWhileUploading: true, + body: { + failureCode: "upload_aborted", + reason: "The Content page gave up this upload.", + attemptId: "attempt-1", + uploadGenerationId: "generation-1", + }, + }); + } + + /** Each read takes the next value, and the last one stays. */ + function readStates( + states: Array>, + claims: Array | null> = [null], + ) { + const next = (values: T[]) => + values.length > 1 ? values.shift() : values[0]; + mockReadAppState.mockImplementation(async (key: string) => + key === "recording-media-verification-rec-1" + ? null + : key === claimKey + ? (next(claims) ?? null) + : next(states), + ); + } + + it("changes nothing while a final chunk holds the upload", async () => { + mockSelectRows.rows = [uploading]; + readStates([uploadState], [claim]); + + await expect(contentCancel()).resolves.toEqual({ + ok: true, + recordingId: "rec-1", + phase: "finishing", + chunksCleared: 0, + }); + + expect(mockCompareAndSetAppState).not.toHaveBeenCalled(); + expect(mockCompareAndSetManyAppState).not.toHaveBeenCalled(); + expect(mockDb.update).not.toHaveBeenCalled(); + expect(mockDeleteRecordingChunks).not.toHaveBeenCalled(); + }); + + it("changes nothing once a final chunk claims the upload between the cancel's read and its own claim", async () => { + mockSelectRows.rows = [uploading]; + readStates( + [uploadState, { ...uploadState, finalChunkClaimedAt: claim.claimedAt }], + [null, claim], + ); + mockCompareAndSetManyAppState.mockResolvedValueOnce(false); + + await expect(contentCancel()).resolves.toEqual({ + ok: true, + recordingId: "rec-1", + phase: "finishing", + chunksCleared: 0, + }); + + expect(mockCompareAndSetManyAppState).toHaveBeenCalledTimes(1); + expect(mockDb.update).not.toHaveBeenCalled(); + }); + + it("cancels an upload whose final chunk claim is older than the chunk's longest run", async () => { + mockSelectRows.rows = [uploading]; + readStates( + [{ ...uploadState, finalChunkClaimedAt: "2026-10-24T00:00:00.000Z" }], + [{ claimId: "claim-1", claimedAt: "2026-10-24T00:00:00.000Z" }], + ); + + await expect(contentCancel()).resolves.toMatchObject({ + ok: true, + recordingId: "rec-1", + chunksCleared: 2, + }); + + expect(mockUpdateSets).toEqual( + expect.arrayContaining([expect.objectContaining({ status: "failed" })]), + ); + }); + + it("cancels an upload whose final chunk failed, whatever its state said meanwhile", async () => { + mockSelectRows.rows = [uploading]; + readStates([{ ...uploadState, status: "processing" }]); + + await expect(contentCancel()).resolves.toMatchObject({ + ok: true, + recordingId: "rec-1", + chunksCleared: 2, + }); + + expect(mockUpdateSets).toEqual( + expect.arrayContaining([expect.objectContaining({ status: "failed" })]), + ); + }); + + it("leaves a recording that finalization moved on before verification starts", async () => { + mockSelectRows.rows = [{ ...uploading, status: "processing" }]; + readStates([uploadState]); + + await expect(contentCancel()).resolves.toEqual({ + ok: true, + recordingId: "rec-1", + phase: "finished", + chunksCleared: 0, + }); + + expect(mockCompareAndSetManyAppState).not.toHaveBeenCalled(); + expect(mockDb.update).not.toHaveBeenCalled(); + }); + + it("still cancels while the final chunk isn't in", async () => { + mockSelectRows.rows = [uploading]; + readStates([uploadState]); + + await expect(contentCancel()).resolves.toMatchObject({ + ok: true, + recordingId: "rec-1", + }); + + expect(mockEqCalls).toContainEqual(["recordings.status", "uploading"]); + expect(mockUpdateSets).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: "failed", + failureCode: "upload_aborted", + }), + ]), + ); + }); + }); + it("aborts provider storage before deleting a resumable session", async () => { mockSelectRows.rows = [ { diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.ts b/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.ts index 2aaf7de17bc..47b21f64f4d 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/abort.post.ts @@ -15,6 +15,7 @@ import { } from "h3"; import { getDb, schema } from "../../../../db/index.js"; +import { readUploadPhase } from "../../../../lib/final-chunk-claim.js"; import { mediaVerificationStateKey } from "../../../../lib/media-verification-state.js"; import { reconcileMeetingOnRecordingFailed } from "../../../../lib/reconcile-meeting-on-finalize.js"; import { @@ -78,6 +79,16 @@ export async function handleAbortRecordingUpload( ownerEmail?: string; orgId?: string; authUserId?: string; + /** The abort request, from a caller that reads its own body. */ + body?: Record; + /** + * Change nothing unless the upload is `uploading` or `parked` + * (`readUploadPhase`), and answer the `phase` it found. A final chunk + * claims the upload before Clips takes its bytes, changing the upload + * state, so the claim below fails for an abort that read the state + * first, and the abort reads the phase again. + */ + onlyWhileUploading?: boolean; }, ) { const recordingId = @@ -98,7 +109,9 @@ export async function handleAbortRecordingUpload( orgId: context.orgId, authUserId: context.authUserId, })); - const body = (await readBody(event).catch(() => null)) as { + const body = (override?.body ?? + // coercion-ok: an unreadable abort report still aborts, with its reason recorded as unknown. + (await readBody(event).catch(() => null))) as { reason?: unknown; failureCode?: unknown; failureStage?: unknown; @@ -298,6 +311,17 @@ export async function handleAbortRecordingUpload( chunksCleared: 0, }; } + // Read after the upload state the claim below expects, so a final + // chunk's claim either shows here or fails that claim. + const leftAlone = async () => { + if (override?.onlyWhileUploading !== true) return null; + const standing = await readUploadPhase(recordingId, ownerEmail); + return standing?.phase === "uploading" || standing?.phase === "parked" + ? null + : { ok: true, recordingId, phase: standing?.phase, chunksCleared: 0 }; + }; + const untouched = await leftAlone(); + if (untouched) return untouched; let preserveRecoveryState = isStoredButUnservableFinalizeError(failureReason) || @@ -428,6 +452,8 @@ export async function handleAbortRecordingUpload( chunksCleared: 0, }; } + const untouchedLatest = await leftAlone(); + if (untouchedLatest) return untouchedLatest; transitionExisting = latest; transitionUploadStateSnapshot = diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.test.ts b/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.test.ts index db8901cbda7..534acd5c803 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.test.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.test.ts @@ -1,4 +1,5 @@ import { MAX_UPLOAD_BYTES } from "@shared/upload-limits.js"; +import type { H3Event } from "h3"; import { beforeEach, describe, expect, it, vi } from "vitest"; const RECORDING_TOO_LARGE_REASON = `Recording exceeds the ${Math.round(MAX_UPLOAD_BYTES / (1024 * 1024))} MB size limit. Please record a shorter clip.`; @@ -46,6 +47,7 @@ const mockGetAppConfig = vi.hoisted(() => vi.fn(() => ({ clips: { hostedStorage: { maxClipBytes: 100_000_000 } } })), ); const mockFinalizeRun = vi.hoisted(() => vi.fn()); +const mockRefuseContentUploadChunk = vi.hoisted(() => vi.fn()); const mockUpdateSets = vi.hoisted(() => [] as Record[]); const mockUpdateRows = vi.hoisted(() => ({ current: [] as Array<{ @@ -156,6 +158,9 @@ vi.mock("../../../../db/index.js", () => ({ }, })); +vi.mock("../../../../lib/content-uploads.js", () => ({ + refuseContentUploadChunk: mockRefuseContentUploadChunk, +})); vi.mock("../../../../lib/recording-upload-state.js", async (importOriginal) => { const actual = await importOriginal< @@ -360,6 +365,42 @@ describe("/api/uploads/:recordingId/chunk route", () => { ); }); + it("refuses a chunk for an upload from a Content page before reading it", async () => { + mockRefuseContentUploadChunk.mockRejectedValueOnce( + Object.assign(new Error("This upload came from a Content page."), { + statusCode: 409, + }), + ); + setRequest({ + query: { index: "0", total: "2", isFinal: "0", mimeType: "video/webm" }, + body: new Uint8Array([1, 2, 3]), + }); + + await expect(handler({} as H3Event)).rejects.toMatchObject({ + statusCode: 409, + }); + expect(mockRefuseContentUploadChunk).toHaveBeenCalledWith("rec-1"); + expect(mockReadRawBody).not.toHaveBeenCalled(); + expect(mockWriteAppState).not.toHaveBeenCalled(); + }); + + it("leaves an upload's page binding to Content's own chunk route", async () => { + setRequest({ + query: { index: "0", total: "2", isFinal: "0", mimeType: "video/webm" }, + body: new Uint8Array([1, 2, 3]), + }); + + await handleRecordingChunk({} as H3Event, { + recordingId: "rec-1", + ownerEmail: "owner@example.com", + orgId: "org-1", + fromContentPage: true, + }); + + expect(mockRefuseContentUploadChunk).not.toHaveBeenCalled(); + expect(mockReadRawBody).toHaveBeenCalled(); + }); + it("rejects chunk indices that cannot fit the exact scratch-key contract", async () => { setRequest({ query: { diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.ts b/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.ts index 10f859559a3..a25f66194d5 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/chunk.post.ts @@ -27,6 +27,7 @@ import { import finalizeRecording from "../../../../../actions/finalize-recording.js"; import { getDb, schema } from "../../../../db/index.js"; +import { refuseContentUploadChunk } from "../../../../lib/content-uploads.js"; import { debugLog } from "../../../../lib/debug.js"; import { mediaVerificationStateKey } from "../../../../lib/media-verification-state.js"; import { reconcileMeetingOnRecordingFailedBestEffort } from "../../../../lib/reconcile-meeting-on-finalize.js"; @@ -511,6 +512,8 @@ export async function handleRecordingChunk( ownerEmail?: string; orgId?: string; authUserId?: string; + /** Content's own chunk route, which checked the upload's page binding. */ + fromContentPage?: boolean; }, ) { const recordingId = @@ -628,6 +631,9 @@ export async function handleRecordingChunk( }); throw createError({ statusCode: 404, message: "Recording not found" }); } + if (!override?.fromContentPage) { + await refuseContentUploadChunk(recordingId); + } const maxUploadBytes = existing.storageMode === "hosted" diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/content-cancel.post.ts b/templates/clips/server/routes/api/uploads/[recordingId]/content-cancel.post.ts new file mode 100644 index 00000000000..7b3a03ca77e --- /dev/null +++ b/templates/clips/server/routes/api/uploads/[recordingId]/content-cancel.post.ts @@ -0,0 +1,6 @@ +// guard:allow-api-route - a document's uploads are a ticketed byte transport from the paired Content's pages, not a CRUD API. +import { defineEventHandler } from "h3"; + +import { handleContentClipCancel } from "../../../../lib/content-clip-uploads.js"; + +export default defineEventHandler(handleContentClipCancel); diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/content-chunk.post.ts b/templates/clips/server/routes/api/uploads/[recordingId]/content-chunk.post.ts new file mode 100644 index 00000000000..76b14d9a93c --- /dev/null +++ b/templates/clips/server/routes/api/uploads/[recordingId]/content-chunk.post.ts @@ -0,0 +1,6 @@ +// guard:allow-api-route - a document's uploads are a ticketed byte transport from the paired Content's pages, not a CRUD API. +import { defineEventHandler } from "h3"; + +import { handleContentClipChunk } from "../../../../lib/content-clip-uploads.js"; + +export default defineEventHandler(handleContentClipChunk); diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.test.ts b/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.test.ts index 42fa3d88442..22bc6336f2b 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.test.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.test.ts @@ -1,3 +1,4 @@ +import type { H3Event } from "h3"; import { beforeEach, describe, expect, it, vi } from "vitest"; const mockRenewUploadLease = vi.hoisted(() => vi.fn()); @@ -316,6 +317,34 @@ describe("/api/uploads/:recordingId/resume route", () => { expect(mockDb.update).toHaveBeenCalledOnce(); }); + it("drops where an earlier final chunk left the upload when it takes chunks again", async () => { + mockSelectRows.rows = [ + { + id: "rec-1", + status: "failed", + failureReason: + "Upload was interrupted. The local recording is safe; retry from the Clips desktop app.", + uploadGenerationId: "generation-1", + }, + ]; + mockReadAppState.mockResolvedValueOnce({ + recordingId: "rec-1", + status: "failed", + uploadGenerationId: "generation-1", + finalChunkOutcome: "finished", + }); + + await expect(handler({} as H3Event)).resolves.toEqual( + expect.objectContaining({ resumable: true, status: "uploading" }), + ); + expect(mockCompareAndSetAppState.mock.calls[0]?.[2]).toMatchObject({ + status: "uploading", + }); + expect(mockCompareAndSetAppState.mock.calls[0]?.[2]).not.toHaveProperty( + "finalChunkOutcome", + ); + }); + it("accepts an interruption with its detailed retryable reason", async () => { mockSelectRows.rows = [ { diff --git a/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.ts b/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.ts index 8d8ed4039dd..a2b7cffdcec 100644 --- a/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.ts +++ b/templates/clips/server/routes/api/uploads/[recordingId]/resume.get.ts @@ -32,6 +32,7 @@ import { import { isRetryableUploadInterruption } from "../../../../../shared/upload-interruption.js"; import { getDb, schema } from "../../../../db/index.js"; +import { withoutFinalChunkOutcome } from "../../../../lib/final-chunk-claim.js"; import { getUploadRecoveryPolicy } from "../../../../lib/recording-policy.js"; import { listRecordingChunkKeys, @@ -332,7 +333,7 @@ export default defineEventHandler(async (event: H3Event) => { uploadGenerationId: generationId, }; const claimedUploadState: Record = { - ...uploadState, + ...withoutFinalChunkOutcome(uploadState), recordingId, status: "uploading", failureReason: null, diff --git a/templates/clips/shared/content-clip-contract.ts b/templates/clips/shared/content-clip-contract.ts index d82cb13c2c6..a70655ead75 100644 --- a/templates/clips/shared/content-clip-contract.ts +++ b/templates/clips/shared/content-clip-contract.ts @@ -14,6 +14,10 @@ // extraClaims: { scope, document_id, recording_id, name? }, // }) // +// An `upload` token names no recording: Clips makes the recording when the +// upload starts. It may carry `client_ip`, the uploader's address as Content's +// edge saw it, which Clips' free storage counts per network. +// // No org domain and no `preferGlobalSecret`: either would let core sign with // another secret. That gives an HS256 token whose `iss` is Content's app URL, // which must equal the `content_app` Clips stored for the document's Clip. The @@ -30,6 +34,11 @@ export const CONTENT_CLIP_SCOPES = { comment: "content-clip:comment", /** A signed-out reader of a public document: playback and the comment list. */ publicRead: "content-clip:public-read", + /** + * A signed-in editor of the document: reading their Clips storage, and + * starting an upload that becomes a Clip bound to the document. + */ + upload: "content-clip:upload", } as const; export type ContentClipScope = @@ -43,6 +52,8 @@ export const CONTENT_CLIP_CLAIMS = { recordingId: "recording_id", /** Optional display name of a signed-in reader, used on their comments. */ viewerName: "name", + /** Optional, on an upload token: the uploader's IP address. */ + clientIp: "client_ip", } as const; /** Longest token lifetime the routes accept; Content mints one-minute tokens. */ @@ -89,3 +100,120 @@ export const CONTENT_CLIP_GRANT_EXPIRED = "content_grant_expired"; * `CONTENT_CLIPS_PEER_SECRET`: unset, too short, or the same as `A2A_SECRET`. */ export const CONTENT_CLIP_NOT_CONFIGURED = "content_clips_not_configured"; + +/** + * The video types an upload from a document may be. Content sends the chunks + * with no `Content-Type`, so the browser makes no CORS preflight; the type + * travels in the chunk URL Clips returns. A cancel goes to the cancel URL the + * same way, with no body. + */ +export const CONTENT_CLIP_UPLOAD_MIME_TYPES = [ + "video/mp4", + "video/webm", + "video/quicktime", +] as const; + +export type ContentClipUploadMimeType = + (typeof CONTENT_CLIP_UPLOAD_MIME_TYPES)[number]; + +/** The longest file name an upload start takes; Clips titles the Clip with it. */ +export const CONTENT_CLIP_UPLOAD_FILE_NAME_CHARS = 255; + +/** + * Where an editor's uploads would go, from `GET /api/content-clips/upload-status` + * and with every refused upload start. Content shows it as it is. + * + * - `ready`: the editor's workspace has its own storage (`builder` or + * `custom`), or a local development Clips keeps chunks in SQL (`local`). + * - `free`: Clips' free hosted storage takes `remaining` more of `limit`. + * - `used_up`: the free clips are used up; connecting storage continues. + * - `none`: no storage and no free clips. `reason` is the free storage's + * refusal: `disabled` when this Clips offers none, or why this editor + * can't have it now. + * - `account_required`: the email has no Clips account with a workspace yet. + */ +export type ContentClipStorage = + | { state: "ready"; provider: "builder" | "custom" | "local" } + | { state: "free"; remaining: number; limit: number; maxClipBytes: number } + | { state: "used_up"; limit: number } + | { state: "none"; reason: string } + | { state: "account_required" }; + +/** + * What `POST /api/uploads/:recordingId/content-cancel` did, as `outcome`. + * + * - `cancelled` (200): the upload is failed in Clips. Its free clip is + * released once Clips confirms the upload's cleanup. + * - `already_finished` (409): Clips has the whole file and is making the + * Clip, or made it; `status` says which. Nothing changed. + * - `upload_finishing` (409): Clips is taking the final chunk and doesn't + * know yet whether it will make the Clip. Nothing changed; cancel again + * after a few seconds, and the answer is `cancelled` or + * `already_finished`. + * + * Any other answer is a refusal with only `error`: the upload wasn't + * cancelled, and Clips fails it once its lease runs out. + */ +export const CONTENT_CLIP_CANCEL_OUTCOME = { + cancelled: "cancelled", + alreadyFinished: "already_finished", + finishing: "upload_finishing", +} as const; + +export type ContentClipCancelAnswer = + | { + outcome: typeof CONTENT_CLIP_CANCEL_OUTCOME.cancelled; + recordingId: string; + } + | { + outcome: typeof CONTENT_CLIP_CANCEL_OUTCOME.alreadyFinished; + recordingId: string; + status: "processing" | "ready"; + } + | { + outcome: typeof CONTENT_CLIP_CANCEL_OUTCOME.finishing; + recordingId: string; + }; + +/** + * The `outcome` a final chunk posted to the chunk URL answers, 409, while + * Clips is still taking an earlier send of it: nothing changed, and the same + * chunk can go again in a few seconds, until the earlier send's claim ends + * (`FINAL_CHUNK_CLAIM_TTL_MS`, two minutes). A final chunk sent again once + * Clips took the first answers with that one's success. + */ +export const CONTENT_CLIP_CHUNK_FINISHING = + CONTENT_CLIP_CANCEL_OUTCOME.finishing; + +/** The `code` of an upload start Clips refused, with `storage` in the body. */ +export const CONTENT_CLIP_UPLOAD_REFUSED = { + /** No storage takes the file: see `storage`. */ + storageRequired: "storage_required", + /** The file is larger than the storage takes; `maxBytes` says how large. */ + tooLarge: "upload_too_large", + /** The editor has too many uploads in progress. */ + tooMany: "too_many_uploads", + /** + * A start repeated with an `uploadKey` whose upload already has the whole + * file: `recordingId` names the Clip and `status` is `processing` or + * `ready`. Nothing was made. + */ + alreadyFinished: "upload_already_finished", + /** + * A start repeated with an `uploadKey` while the first start is still + * making its recording, or while Clips takes its final chunk, which can + * still fail and leave the upload open. Repeat it again in a few seconds. + * Once a start with no recording failed without saying it left nothing, or + * has gone about ten minutes without one, the next repeat takes it over and + * starts it again with the same recording id. + */ + uploadStarting: "upload_starting", + /** + * A start repeated with an `uploadKey` that can't continue, and nothing was + * made: the key's upload ended (cancelled, or failed, a start that failed + * after making its recording included), so the file starts again under a + * new key. Also when the upload is parked until storage is connected, + * began in another block, or was for a file of another type or size. + */ + keyUsed: "upload_key_used", +} as const;