diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f1e0d5c..14e439ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,13 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Added + +- **`ExternalSubtitleTrack.sourceStreamIndex`, so an external subtitle URL can be a container rather than a sidecar.** An external track's URL may hold several subtitle streams (an MKV with English, English SDH and Spanish), and a host would register one track per stream against that same URL. The sidecar decoder stopped at the container's first subtitle stream and the descriptor carried no index, so every such track decoded that same stream and the host got three selectable tracks rendering identical cues. The new field names the stream to decode as an absolute `AVStream` index inside the container, matching the convention that embedded track ids are stream indices; nil keeps decoding the first subtitle stream. An index that is out of range or names a non-subtitle stream fails the decode rather than falling back to the first subtitle stream, because a silent fallback is indistinguishable from the behaviour the index exists to escape. Reported by edde746. (#266) + +### Changed + +- **External tracks sharing a container are now filled from a single pass over it.** Each load-declared external track used to be decoded by its own whole-file read, so three tracks pointing at one MKV meant three full downloads at load, and `AVDISCARD_ALL` cannot shorten them (a Matroska demuxer reads every discarded byte anyway). Tracks sharing a URL and headers are now decoded together in one pass, one stream decoder per requested stream. Since a pass covering several streams fails as a whole, a failure retries the targets individually, so one host-side index mistake cannot blank the container's other tracks; a store that still could not be filled stays unfinished rather than serving a complete but blank rendition. ## [6.2.1] - 2026-07-30 diff --git a/README.md b/README.md index dd45ba63..4629cff7 100644 --- a/README.md +++ b/README.md @@ -167,6 +167,9 @@ let track = player.addExternalSubtitleTrack( player.selectSubtitleTrack(index: track.id) // Declared at load instead, external tracks also join the native WebVTT renditions (PiP): // LoadOptions(prepareNativeSubtitles: true, externalSubtitles: [ExternalSubtitleTrack(url: srtURL, language: "en")]) +// When the URL is a container holding several subtitle streams, register one track per stream +// with its absolute AVStream index; tracks sharing a URL are decoded in one pass (#266). +ExternalSubtitleTrack(url: mkvURL, name: "Spanish", language: "es", sourceStreamIndex: 3) // Native WebVTT subtitle renditions (subtitles in PiP / AirPlay / external display; opt-in // via LoadOptions.prepareNativeSubtitles, details in docs/formats.md) diff --git a/Sources/AetherEngine/AetherEngine+Subtitles.swift b/Sources/AetherEngine/AetherEngine+Subtitles.swift index 1406ae28..4201638e 100644 --- a/Sources/AetherEngine/AetherEngine+Subtitles.swift +++ b/Sources/AetherEngine/AetherEngine+Subtitles.swift @@ -144,7 +144,8 @@ extension AetherEngine { clearSubtitleDrainTarget(channel: .secondary) // #112 rework activeSecondaryEmbeddedSubtitleStreamIndex = -1 activeSecondaryExternalSubtitleTrackID = index - startSecondarySidecarDecode(url: external.url, httpHeaders: external.httpHeaders) + startSecondarySidecarDecode(url: external.url, httpHeaders: external.httpHeaders, + sourceStreamIndex: external.sourceStreamIndex) return } guard index < Self.externalSubtitleTrackIDBase else { return } @@ -873,7 +874,8 @@ extension AetherEngine { EngineLog.emit("[AetherEngine] external subtitle backfilled from finished store: id=\(id) cues=\(subtitleCues.count)", category: .engine) return } - startSidecarDecode(url: track.url, httpHeaders: track.httpHeaders, externalTrackID: id) + startSidecarDecode(url: track.url, httpHeaders: track.httpHeaders, externalTrackID: id, + sourceStreamIndex: track.sourceStreamIndex) } /// Store lookup for the external backfill: test-hook override first, else the live session's stores. @@ -892,32 +894,54 @@ extension AetherEngine { func startExternalNativeStoreFill(session: HLSVideoEngine) { externalNativeStoreFillTask?.cancel() externalNativeStoreFillTask = nil - var jobs: [(url: URL, headers: [String: String], store: NativeSubtitleCueStore)] = [] - for (ordinal, entry) in nativeSubtitleTrackTable.enumerated() { - // Phase D: OCR entries defer to the selection-time sidecar decode (OCR of a whole - // .sup at load would violate the selection gating). - guard !entry.needsOCR, - let extID = entry.externalID, - let track = externalSubtitleRegistry[extID], - ordinal < session.nativeSubtitleCueStoresForSession.count else { continue } - jobs.append((track.url, - track.httpHeaders ?? loadedOptions.httpHeaders, - session.nativeSubtitleCueStoresForSession[ordinal])) - } + let jobs = Self.externalSubtitleFillJobs( + table: nativeSubtitleTrackTable, + registry: externalSubtitleRegistry, + stores: session.nativeSubtitleCueStoresForSession, + defaultHeaders: loadedOptions.httpHeaders) guard !jobs.isEmpty else { return } externalNativeStoreFillTask = Task.detached(priority: .utility) { [jobs] in for job in jobs { if Task.isCancelled { return } - if let result = try? await SubtitleDecoder.decodeFile(url: job.url, httpHeaders: job.headers) { - job.store.appendCues(result.cues) - job.store.markFinished() - } else { - EngineLog.emit("[AetherEngine] external native store fill failed: \(job.url.lastPathComponent)", category: .engine) - } + await AetherEngine.runExternalSubtitleFill(job: job) } } } + /// #266: fill one container's stores from a single decode pass. A pass covering several streams + /// fails as a whole (an out-of-range index throws), so on failure the targets are retried + /// individually: one host-side index mistake must not blank the container's other tracks. A + /// store that could not be filled stays UNfinished, or the rendition would serve a complete but + /// blank .vtt. + nonisolated static func runExternalSubtitleFill(job: ExternalSubtitleFillJob) async { + if let results = try? await SubtitleDecoder.decodeFile( + url: job.url, httpHeaders: job.headers, + sourceStreamIndices: job.targets.map(\.streamIndex) + ) { + for (target, result) in zip(job.targets, results) { + target.store.appendCues(result.cues) + target.store.markFinished() + } + return + } + guard job.targets.count > 1 else { + EngineLog.emit("[AetherEngine] external native store fill failed: \(job.url.lastPathComponent)", category: .engine) + return + } + EngineLog.emit("[AetherEngine] external native store fill: shared pass over \(job.url.lastPathComponent) failed, retrying \(job.targets.count) targets individually", category: .engine) + for target in job.targets { + if Task.isCancelled { return } + guard let result = try? await SubtitleDecoder.decodeFile( + url: job.url, httpHeaders: job.headers, sourceStreamIndex: target.streamIndex + ) else { + EngineLog.emit("[AetherEngine] external native store fill failed: \(job.url.lastPathComponent) stream=\(target.streamIndex.map(String.init) ?? "auto")", category: .engine) + continue + } + target.store.appendCues(result.cues) + target.store.markFinished() + } + } + /// Unregister an external track: delist + drop the registry entry; an active selection /// (primary or secondary) is cleared. Embedded ids no-op. public func removeExternalSubtitleTrack(id: Int) { @@ -937,7 +961,8 @@ extension AetherEngine { /// track id (if any) to publish as active. Also clears the pump-tap overlay stream so a prior /// tap-fed selection stops forwarding into the sidecar's cues (latent pre-#88 bug: the tap /// forward-guard matched the stale index and kept appending). - func startSidecarDecode(url: URL, httpHeaders: [String: String]?, externalTrackID: Int?) { + func startSidecarDecode(url: URL, httpHeaders: [String: String]?, externalTrackID: Int?, + sourceStreamIndex: Int32? = nil) { cancelSidecarTask() // Sidecar replaces any active embedded stream. clearSubtitleDrainTarget(channel: .primary) // #112 rework @@ -959,7 +984,8 @@ extension AetherEngine { do { result = try await SubtitleDecoder.decodeFile( url: url, httpHeaders: effectiveHeaders, - preserveASSMarkup: preserveASS + preserveASSMarkup: preserveASS, + sourceStreamIndex: sourceStreamIndex ) } catch { EngineLog.emit("[AetherEngine] sidecar decode failed: \(error)", category: .engine) @@ -998,7 +1024,8 @@ extension AetherEngine { } /// Shared secondary sidecar-decode start (#88): the pre-#88 selectSecondarySidecarSubtitle body. - func startSecondarySidecarDecode(url: URL, httpHeaders: [String: String]?) { + func startSecondarySidecarDecode(url: URL, httpHeaders: [String: String]?, + sourceStreamIndex: Int32? = nil) { loadedSecondarySidecarURL = url isSecondarySubtitleActive = true secondarySubtitleCues = [] @@ -1010,7 +1037,8 @@ extension AetherEngine { let result: SidecarDecodeResult do { // Secondary is plain text only (never drives libass, mirroring embedded secondary #47). - result = try await SubtitleDecoder.decodeFile(url: url, httpHeaders: effectiveHeaders) + result = try await SubtitleDecoder.decodeFile( + url: url, httpHeaders: effectiveHeaders, sourceStreamIndex: sourceStreamIndex) } catch { EngineLog.emit("[AetherEngine] secondary sidecar decode failed: \(error)", category: .engine) await MainActor.run { @@ -1573,6 +1601,38 @@ extension AetherEngine { table.firstIndex { $0.sourceStreamIndex == id || $0.externalID == id } } + /// #266: group the load-declared external tracks into one fill job per container, so a URL + /// backing several tracks (an MKV with three subtitle streams) is read once instead of once per + /// track. Headers are part of the grouping key: differing auth means differing requests. + /// Duplicate registrations of one stream stay separate targets, both stores get the cues. + /// Ordering is by first appearance in the table, so the jobs are deterministic. + nonisolated static func externalSubtitleFillJobs( + table: [NativeSubtitleTrackEntry], + registry: [Int: ExternalSubtitleTrack], + stores: [NativeSubtitleCueStore], + defaultHeaders: [String: String] + ) -> [ExternalSubtitleFillJob] { + struct Key: Hashable { + let url: URL + let headers: [String: String] + } + var order: [Key] = [] + var targetsByKey: [Key: [ExternalSubtitleFillJob.Target]] = [:] + for (ordinal, entry) in table.enumerated() { + // Phase D: OCR entries defer to the selection-time sidecar decode (OCR of a whole + // .sup at load would violate the selection gating). + guard !entry.needsOCR, let extID = entry.externalID, + let track = registry[extID], ordinal < stores.count else { continue } + let key = Key(url: track.url, headers: track.httpHeaders ?? defaultHeaders) + if targetsByKey[key] == nil { order.append(key) } + targetsByKey[key, default: []].append( + .init(streamIndex: track.sourceStreamIndex, store: stores[ordinal])) + } + return order.map { + ExternalSubtitleFillJob(url: $0.url, headers: $0.headers, targets: targetsByKey[$0] ?? []) + } + } + /// Phase D: bitmap tracks eligible for an OCR-fed rendition. VOD only; embedded entries carry /// their source stream index (the worker's packet-store key), external .sup entries their /// synthetic id (the sidecar OCR fill key). diff --git a/Sources/AetherEngine/AetherEngine.swift b/Sources/AetherEngine/AetherEngine.swift index aea15811..d051a9b4 100644 --- a/Sources/AetherEngine/AetherEngine.swift +++ b/Sources/AetherEngine/AetherEngine.swift @@ -1203,6 +1203,20 @@ public final class AetherEngine: ObservableObject { } var nativeSubtitleTrackTable: [NativeSubtitleTrackEntry] = [] + /// #266: one pass over one container, filling every native store whose external track points at + /// it. Tracks that share a URL and headers collapse into a single job, so a container holding + /// several subtitle streams is fetched once rather than once per registered track. + struct ExternalSubtitleFillJob: Sendable { + struct Target: Sendable { + /// Absolute AVStream index in this container, nil for its first subtitle stream. + let streamIndex: Int32? + let store: NativeSubtitleCueStore + } + let url: URL + let headers: [String: String] + let targets: [Target] + } + /// Native WebVTT rendition store for the in-band CEA-608 track (#98). The CC tap feeds it (via /// `updateClosedCaptionCues`) so 608 captions ride a native AVKit-selectable rendition and /// survive PiP / AirPlay, not just the overlay. Nil when there is no 608 track or native diff --git a/Sources/AetherEngine/Decoder/SubtitleDecoder.swift b/Sources/AetherEngine/Decoder/SubtitleDecoder.swift index 85cba729..398c0a11 100644 --- a/Sources/AetherEngine/Decoder/SubtitleDecoder.swift +++ b/Sources/AetherEngine/Decoder/SubtitleDecoder.swift @@ -6,6 +6,10 @@ import Libavutil enum SubtitleDecoderError: Error { case openFailed(code: Int32) case noSubtitleStream + /// #266: a requested absolute stream index is out of range or not a subtitle stream. Reported + /// rather than falling back to the first subtitle stream, since a silent fallback is + /// indistinguishable from the pre-#266 behaviour the index exists to escape. + case streamIndexNotSubtitle(index: Int32) case noDecoder case codecOpenFailed(code: Int32) } @@ -24,11 +28,49 @@ enum SubtitleDecoder { /// Decode every cue from the subtitle file at `url`, cancellable via Task.cancel(). /// When preserveASSMarkup is true, ASS/SSA cues carry the raw libavcodec event line /// (ReadOrder,Layer,Style,...,Text) so ASSScriptBuilder can restyle them; no effect on SRT/VTT. + /// `sourceStreamIndex` (#266) is an ABSOLUTE AVStream index inside the container at `url`, for + /// URLs that are containers holding several subtitle streams; nil decodes the container's first + /// subtitle stream (the pre-#266 behaviour). An index that is out of range or names a + /// non-subtitle stream throws `streamIndexNotSubtitle`. static func decodeFile( url: URL, httpHeaders: [String: String] = [:], - preserveASSMarkup: Bool = false + preserveASSMarkup: Bool = false, + sourceStreamIndex: Int32? = nil ) async throws -> SidecarDecodeResult { + let results = try await decode( + url: url, httpHeaders: httpHeaders, preserveASSMarkup: preserveASSMarkup, + requested: sourceStreamIndex.map { [$0] } + ) + guard let only = results.first else { throw SubtitleDecoderError.noSubtitleStream } + return only + } + + /// #266: decode several subtitle streams from ONE pass over the container, so a host that + /// registered one external track per embedded stream pays a single fetch instead of one per + /// track. The result is POSITIONAL: element i holds the cues for `sourceStreamIndices[i]`, so a + /// caller maps it straight back onto its own targets without resolving what a nil entry became. + /// A repeated index is decoded once and returned at each of its positions. + static func decodeFile( + url: URL, + httpHeaders: [String: String] = [:], + preserveASSMarkup: Bool = false, + sourceStreamIndices: [Int32?] + ) async throws -> [SidecarDecodeResult] { + guard !sourceStreamIndices.isEmpty else { return [] } + return try await decode( + url: url, httpHeaders: httpHeaders, preserveASSMarkup: preserveASSMarkup, + requested: sourceStreamIndices + ) + } + + /// `requested` nil means "the container's first subtitle stream", the pre-#266 single-stream path. + private static func decode( + url: URL, + httpHeaders: [String: String], + preserveASSMarkup: Bool, + requested: [Int32?]? + ) async throws -> [SidecarDecodeResult] { // Task.cancel() does NOT propagate into detached tasks (isCancelled inside always false). // Bridge cancellation explicitly via CancelFlag so the decode loop + AVIO reader abort promptly. let token = CancelFlag() @@ -36,7 +78,7 @@ enum SubtitleDecoder { try await Task.detached(priority: .userInitiated) { try decodeFileSync( url: url, httpHeaders: httpHeaders, - preserveASSMarkup: preserveASSMarkup, cancel: token + preserveASSMarkup: preserveASSMarkup, requested: requested, cancel: token ) }.value } onCancel: { @@ -71,12 +113,240 @@ enum SubtitleDecoder { } } + // MARK: - Per-stream decode state + + /// One decoder plus its accumulated cues, so #266 can run several streams off a single read + /// loop. Every field here used to be a local in `decodeFileSync`; the timing anchor, the ASS + /// PlayRes and the open-image bookkeeping are all per-stream and must not be shared. + private final class StreamDecode { + let streamIndex: Int32 + let codecCtx: UnsafeMutablePointer + let tbSec: Double + let keepMarkup: Bool + let assPlayRes: CGSize + let assHeader: String? + let codedWidth: Int + let codedHeight: Int + + var cues: [SubtitleCue] = [] + var nextID = 0 + /// Indices of image cues still "open" (PGS-style: ended by the next composition event). + var pendingImageCueIndices: [Int] = [] + var lastPktPTS: Double = 0 // PTS anchor for flush events that have no packet of their own + + init(streamIndex: Int32, stream: UnsafeMutablePointer, preserveASSMarkup: Bool) throws { + guard let codecpar = stream.pointee.codecpar else { + throw SubtitleDecoderError.streamIndexNotSubtitle(index: streamIndex) + } + + // ASS/SSA script header is in codec extradata (mirrors Demuxer.trackInfo for embedded tracks). + // Only surfaced under preserveASSMarkup; the raw event-line path is the only consumer. + let codecID = codecpar.pointee.codec_id + let isASS = codecID == AV_CODEC_ID_ASS || codecID == AV_CODEC_ID_SSA + let keepMarkup = preserveASSMarkup && isASS + var assHeader: String? = nil + var assPlayRes = SubtitleRectText.defaultASSPlayRes + if let extradata = codecpar.pointee.extradata, codecpar.pointee.extradata_size > 0 { + let bytes = Data(bytes: extradata, count: Int(codecpar.pointee.extradata_size)) + // Strip NUL bytes: extradata is often NUL-terminated; libass parses C-string-style and a NUL hides everything after it. + let header = String(data: bytes, encoding: .utf8)? + .replacingOccurrences(of: "\0", with: "") + if keepMarkup { assHeader = header } + // #233: a real ASS script declares the space its \pos coordinates live in; without one + // the line came from libavcodec's own conversion and uses the 384x288 default. + if let header, let declared = SubtitleRectText.playRes(fromASSHeader: header) { + assPlayRes = declared + } + } + + guard let codec = avcodec_find_decoder(codecpar.pointee.codec_id) else { + throw SubtitleDecoderError.noDecoder + } + guard let ctx = avcodec_alloc_context3(codec) else { + throw SubtitleDecoderError.codecOpenFailed(code: -1) + } + // Throwing before `codecCtx` is assigned skips deinit, so unwind the context by hand. + var local: UnsafeMutablePointer? = ctx + let paramsRet = avcodec_parameters_to_context(ctx, codecpar) + guard paramsRet >= 0 else { + avcodec_free_context(&local) + throw SubtitleDecoderError.codecOpenFailed(code: paramsRet) + } + let openRet = avcodec_open2(ctx, codec, nil) + guard openRet >= 0 else { + avcodec_free_context(&local) + throw SubtitleDecoderError.codecOpenFailed(code: openRet) + } + + self.streamIndex = streamIndex + self.codecCtx = ctx + let timeBase = stream.pointee.time_base + self.tbSec = Double(timeBase.num) / Double(timeBase.den) + self.keepMarkup = keepMarkup + self.assPlayRes = assPlayRes + self.assHeader = assHeader + self.codedWidth = Int(codecpar.pointee.width) + self.codedHeight = Int(codecpar.pointee.height) + } + + deinit { + var local: UnsafeMutablePointer? = codecCtx + avcodec_free_context(&local) + } + + // Under preserveASSMarkup: keep raw ASS event line (ASSScriptBuilder re-stamps timing); otherwise plain text. + private func line(for rect: UnsafeMutablePointer) -> String? { + keepMarkup ? SubtitleRectText.rawASSLine(for: rect) : SubtitleRectText.plainText(for: rect) + } + + func decode(packet pkt: UnsafeMutablePointer) { + var sub = AVSubtitle() + var gotSub: Int32 = 0 + let ret = avcodec_decode_subtitle2(codecCtx, &sub, &gotSub, pkt) + guard ret >= 0, gotSub != 0 else { return } + + let pktPTS = pkt.pointee.pts == Int64.min + ? 0.0 + : Double(pkt.pointee.pts) * tbSec + lastPktPTS = pktPTS + let startOffset = Double(sub.start_display_time) / 1000.0 + let endOffset: Double + if sub.end_display_time > 0 { + endOffset = Double(sub.end_display_time) / 1000.0 + } else if pkt.pointee.duration > 0 { + endOffset = Double(pkt.pointee.duration) * tbSec + } else { + endOffset = 5.0 + } + let startTime = pktPTS + startOffset + let endTime = pktPTS + endOffset + + // Bitmap subtitles (external .sup / PGS sidecars, FFmpegBuild >= 2.1.3 sup demuxer): + // a composition usually carries end_display_time == 0 and is ended by the NEXT + // composition event (a new set or a clear packet), so clamp any still-open image + // cues to this packet's PTS before appending the new ones. The 5 s fallback above + // only survives for a final composition with no successor. + for idx in pendingImageCueIndices where cues[idx].startTime < pktPTS && cues[idx].endTime > pktPTS { + let open = cues[idx] + cues[idx] = open.with(endTime: pktPTS) + } + pendingImageCueIndices.removeAll() + + var lines: [String] = [] + var images: [SubtitleImage] = [] + var styledBodies: [SubtitleCue.Body] = [] + var placement: SubtitleTextPlacement? + if sub.num_rects > 0, let rects = sub.rects { + for i in 0.. startTime { + append(startTime: startTime, endTime: endTime, body: .text(merged), placement: placement) + } + if endTime > startTime { + for body in styledBodies { + append(startTime: startTime, endTime: endTime, body: body, placement: placement) + } + } + if endTime > startTime { + for image in images { + pendingImageCueIndices.append(cues.count) + append(startTime: startTime, endTime: endTime, body: .image(image), placement: nil) + } + } + } + + /// Flush ASS/SSA buffered events (old code decoded one event and discarded it, silently losing the last cue). + /// Flushed events have no packet; use lastPktPTS as the timing anchor. + func flush(cancel: CancelFlag) { + while !cancel.isCancelled { + var flushPkt = AVPacket() + flushPkt.data = nil + flushPkt.size = 0 + var flushSub = AVSubtitle() + var gotFlush: Int32 = 0 + let flushRet = avcodec_decode_subtitle2(codecCtx, &flushSub, &gotFlush, &flushPkt) + guard flushRet >= 0, gotFlush != 0 else { break } + + let startOffset = Double(flushSub.start_display_time) / 1000.0 + let endOffset = flushSub.end_display_time > 0 + ? Double(flushSub.end_display_time) / 1000.0 + : startOffset + 5.0 + var lines: [String] = [] + if flushSub.num_rects > 0, let rects = flushSub.rects { + for i in 0.. startTime { + append(startTime: startTime, endTime: endTime, body: .text(merged), placement: nil) + } + } + } + + private func append(startTime: Double, endTime: Double, + body: SubtitleCue.Body, placement: SubtitleTextPlacement?) { + cues.append(SubtitleCue(id: nextID, startTime: startTime, endTime: endTime, + body: body, placement: placement)) + nextID += 1 + } + + var result: SidecarDecodeResult { + SidecarDecodeResult(cues: cues.sorted { $0.startTime < $1.startTime }, assHeader: assHeader) + } + } + // MARK: - Synchronous core private static func decodeFileSync( url: URL, httpHeaders: [String: String], - preserveASSMarkup: Bool, cancel: CancelFlag - ) throws -> SidecarDecodeResult { + preserveASSMarkup: Bool, requested: [Int32?]?, cancel: CancelFlag + ) throws -> [SidecarDecodeResult] { let isHTTP = url.scheme == "http" || url.scheme == "https" var formatContext: UnsafeMutablePointer? @@ -131,73 +401,33 @@ enum SubtitleDecoder { } // Probe defensively; sidecars usually have one stream at index 0 but containers can have extras. - var subStreamIndex: Int = -1 - for i in 0..= 0, - let stream = fmt.pointee.streams[subStreamIndex], - let codecpar = stream.pointee.codecpar - else { - throw SubtitleDecoderError.noSubtitleStream + guard request >= 0, request < Int32(fmt.pointee.nb_streams), + let stream = fmt.pointee.streams[Int(request)], + let codecpar = stream.pointee.codecpar, + codecpar.pointee.codec_type == AVMEDIA_TYPE_SUBTITLE + else { throw SubtitleDecoderError.streamIndexNotSubtitle(index: request) } + return request } - // ASS/SSA script header is in codec extradata (mirrors Demuxer.trackInfo for embedded tracks). - // Only surfaced under preserveASSMarkup; the raw event-line path is the only consumer. - let codecID = codecpar.pointee.codec_id - let isASS = codecID == AV_CODEC_ID_ASS || codecID == AV_CODEC_ID_SSA - let keepMarkup = preserveASSMarkup && isASS - var assHeader: String? = nil - var assPlayRes = SubtitleRectText.defaultASSPlayRes - if let extradata = codecpar.pointee.extradata, codecpar.pointee.extradata_size > 0 { - let bytes = Data(bytes: extradata, count: Int(codecpar.pointee.extradata_size)) - // Strip NUL bytes: extradata is often NUL-terminated; libass parses C-string-style and a NUL hides everything after it. - let header = String(data: bytes, encoding: .utf8)? - .replacingOccurrences(of: "\0", with: "") - if keepMarkup { assHeader = header } - // #233: a real ASS script declares the space its \pos coordinates live in; without one - // the line came from libavcodec's own conversion and uses the 384x288 default. - if let header, let declared = SubtitleRectText.playRes(fromASSHeader: header) { - assPlayRes = declared + var decodersByStream: [Int32: StreamDecode] = [:] + // Registered after the avformat_close_input defer, so LIFO runs it FIRST: the codec + // contexts go before the format context, matching the pre-#266 defer order. Covers the + // throwing paths below too. + defer { decodersByStream.removeAll() } + for index in resolved where decodersByStream[index] == nil { + guard let stream = fmt.pointee.streams[Int(index)] else { + throw SubtitleDecoderError.streamIndexNotSubtitle(index: index) } - } - - guard let codec = avcodec_find_decoder(codecpar.pointee.codec_id) else { - throw SubtitleDecoderError.noDecoder - } - guard let codecCtx = avcodec_alloc_context3(codec) else { - throw SubtitleDecoderError.codecOpenFailed(code: -1) - } - var localCodecCtx: UnsafeMutablePointer? = codecCtx - defer { avcodec_free_context(&localCodecCtx) } - - let paramsRet = avcodec_parameters_to_context(codecCtx, codecpar) - guard paramsRet >= 0 else { - throw SubtitleDecoderError.codecOpenFailed(code: paramsRet) - } - let openRet = avcodec_open2(codecCtx, codec, nil) - guard openRet >= 0 else { - throw SubtitleDecoderError.codecOpenFailed(code: openRet) - } - - let timeBase = stream.pointee.time_base - let tbSec = Double(timeBase.num) / Double(timeBase.den) - - var cues: [SubtitleCue] = [] - var nextID = 0 - /// Indices of image cues still "open" (PGS-style: ended by the next composition event). - var pendingImageCueIndices: [Int] = [] - var lastPktPTS: Double = 0 // PTS anchor for flush events that have no packet of their own - - // Under preserveASSMarkup: keep raw ASS event line (ASSScriptBuilder re-stamps timing); otherwise plain text. - let lineForRect: (UnsafeMutablePointer) -> String? = { rect in - keepMarkup ? SubtitleRectText.rawASSLine(for: rect) : SubtitleRectText.plainText(for: rect) + decodersByStream[index] = try StreamDecode( + streamIndex: index, stream: stream, preserveASSMarkup: preserveASSMarkup) } while !cancel.isCancelled { @@ -209,175 +439,27 @@ enum SubtitleDecoder { break } - if Int(pkt.pointee.stream_index) != subStreamIndex { - av_packet_unref(pkt) - trackedPacketFree(&pktPtr) - continue - } - - var sub = AVSubtitle() - var gotSub: Int32 = 0 - let ret = avcodec_decode_subtitle2(codecCtx, &sub, &gotSub, pkt) - - if ret >= 0 && gotSub != 0 { - let pktPTS = pkt.pointee.pts == Int64.min - ? 0.0 - : Double(pkt.pointee.pts) * tbSec - lastPktPTS = pktPTS - let startOffset = Double(sub.start_display_time) / 1000.0 - let endOffset: Double - if sub.end_display_time > 0 { - endOffset = Double(sub.end_display_time) / 1000.0 - } else if pkt.pointee.duration > 0 { - endOffset = Double(pkt.pointee.duration) * tbSec - } else { - endOffset = 5.0 - } - let startTime = pktPTS + startOffset - let endTime = pktPTS + endOffset - - // Bitmap subtitles (external .sup / PGS sidecars, FFmpegBuild >= 2.1.3 sup demuxer): - // a composition usually carries end_display_time == 0 and is ended by the NEXT - // composition event (a new set or a clear packet), so clamp any still-open image - // cues to this packet's PTS before appending the new ones. The 5 s fallback above - // only survives for a final composition with no successor. - for idx in pendingImageCueIndices where cues[idx].startTime < pktPTS && cues[idx].endTime > pktPTS { - let open = cues[idx] - cues[idx] = open.with(endTime: pktPTS) - } - pendingImageCueIndices.removeAll() - - var lines: [String] = [] - var images: [SubtitleImage] = [] - var styledBodies: [SubtitleCue.Body] = [] - var placement: SubtitleTextPlacement? - if sub.num_rects > 0, let rects = sub.rects { - for i in 0.. startTime { - cues.append(SubtitleCue( - id: nextID, - startTime: startTime, - endTime: endTime, - body: .text(merged), - placement: placement - )) - nextID += 1 - } - if endTime > startTime { - for body in styledBodies { - cues.append(SubtitleCue( - id: nextID, - startTime: startTime, - endTime: endTime, - body: body, - placement: placement - )) - nextID += 1 - } - } - if endTime > startTime { - for image in images { - pendingImageCueIndices.append(cues.count) - cues.append(SubtitleCue( - id: nextID, - startTime: startTime, - endTime: endTime, - body: .image(image) - )) - nextID += 1 - } - } - } + decodersByStream[pkt.pointee.stream_index]?.decode(packet: pkt) av_packet_unref(pkt) trackedPacketFree(&pktPtr) } - // Flush ASS/SSA buffered events (old code decoded one event and discarded it, silently losing the last cue). - // Flushed events have no packet; use lastPktPTS as the timing anchor. - while !cancel.isCancelled { - var flushPkt = AVPacket() - flushPkt.data = nil - flushPkt.size = 0 - var flushSub = AVSubtitle() - var gotFlush: Int32 = 0 - let flushRet = avcodec_decode_subtitle2(codecCtx, &flushSub, &gotFlush, &flushPkt) - guard flushRet >= 0, gotFlush != 0 else { break } - - let startOffset = Double(flushSub.start_display_time) / 1000.0 - let endOffset = flushSub.end_display_time > 0 - ? Double(flushSub.end_display_time) / 1000.0 - : startOffset + 5.0 - var lines: [String] = [] - if flushSub.num_rects > 0, let rects = flushSub.rects { - for i in 0.. startTime { - cues.append(SubtitleCue( - id: nextID, - startTime: startTime, - endTime: endTime, - body: .text(merged) - )) - nextID += 1 + return resolved.compactMap { decodersByStream[$0]?.result } + } + + private static func firstSubtitleStreamIndex(in fmt: UnsafeMutablePointer) -> Int32? { + for i in 0.. 00:00:02,000\nfirst english line\n\n2\n00:00:03,000 --> 00:00:04,000\nsecond english line\n' > en.srt +/// printf '1\n00:00:01,500 --> 00:00:02,500\nprimera linea espanola\n\n2\n00:00:03,500 --> 00:00:04,500\nsegunda linea espanola\n' > es.srt +/// ffmpeg -f lavfi -i color=c=black:s=16x16:r=1:d=1 -i en.srt -i es.srt \ +/// -map 0:v -map 1:0 -map 2:0 -c:v libx264 -preset ultrafast -crf 51 -pix_fmt yuv420p -c:s srt \ +/// -metadata:s:s:0 language=eng -metadata:s:s:1 language=spa multi-sub.mkv +/// +/// Stream layout: 0 = h264 video, 1 = subrip (eng), 2 = subrip (spa). +enum MultiSubtitleContainerFixture { + + /// Absolute AVStream index of the English subtitle stream (the FIRST subtitle stream). + static let englishStreamIndex: Int32 = 1 + /// Absolute AVStream index of the Spanish subtitle stream (the SECOND subtitle stream). + static let spanishStreamIndex: Int32 = 2 + /// Absolute AVStream index of the video stream: a valid stream that is not a subtitle stream. + static let videoStreamIndex: Int32 = 0 + + static let englishLines = ["first english line", "second english line"] + static let spanishLines = ["primera linea espanola", "segunda linea espanola"] + + /// Write the container to a unique temporary file and return its URL. The caller owns the file; + /// `withFixture` removes it. + static func write() throws -> URL { + guard let data = Data(base64Encoded: base64.joined()) else { + throw CocoaError(.fileReadCorruptFile) + } + let url = FileManager.default.temporaryDirectory + .appendingPathComponent("ae266-multi-sub-\(UUID().uuidString).mkv") + try data.write(to: url) + return url + } + + /// Run `body` against a freshly written container, removing it afterwards. + static func withFixture(_ body: (URL) async throws -> T) async throws -> T { + let url = try write() + defer { try? FileManager.default.removeItem(at: url) } + return try await body(url) + } + + private static let base64 = [ + "GkXfo6NChoEBQveBAULygQRC84EIQoKIbWF0cm9za2FCh4EEQoWBAhhTgGcBAAAAAAAGmxFNm3TAv4SBEIGRTbuLU6uEFUmpZlOs", + "gaFNu4tTq4QWVK5rU6yB8U27jFOrhBJUw2dTrIIB8027jFOrhBxTu2tTrIIGI+wBAAAAAAAAUwAAAAAAAAAAAAAAAAAAAAAAAAAA", + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFUmpZsu/hEQd6VYq", + "17GDD0JATYCNTGF2ZjYyLjEyLjEwMVdBjUxhdmY2Mi4xMi4xMDFzpJDoHqoUxLLf3xWarpxu+IYgRImIQLGUAAAAAAAWVK5rQPy/", + "hHnEsKOuAQAAAAAAAIDXgQFzxYhe/osBX0V8dpyBACK1nIN1bmSIgQCGj1ZfTVBFRzQvSVNPL0FWQ4OBASPjg4Q7msoA4JCwgRC6", + "gRCagQJVsIRVuYEBVe6BAOwBAAAAAAAAAgAAY6KlAULACv/hABVnQsAK2nsBEAAAAwAQAAADACDxImoBAAVozgGXIK4BAAAAAAAA", + "LNeBAnPFiA6M8olhdHoOnIEAIrWcg2VuZ4aLU19URVhUL1VURjiDgRFV7oEArgEAAAAAAAAv14EDc8WIDwtfqsSY/9mcgQAitZyD", + "c3BhiIEAhotTX1RFWFQvVVRGOIOBEVXugQASVMNnQS+/hCx91aNzc6BjwIBnyJpFo4dFTkNPREVSRIeNTGF2ZjYyLjEyLjEwMXNz", + "12PAi2PFiF7+iwFfRXx2Z8iiRaOHRU5DT0RFUkSHlUxhdmM2Mi4yOC4xMDEgbGlieDI2NGfIoUWjiERVUkFUSU9ORIeTMDA6MDA6", + "MDEuMDAwMDAwMDAwAHNz02PAi2PFiA6M8olhdHoOZ8ieRaOHRU5DT0RFUkSHkUxhdmM2Mi4yOC4xMDEgc3J0Z8ihRaOIRFVSQVRJ", + "T05Eh5MwMDowMDowNC4wMDAwMDAwMDAAc3PTY8CLY8WIDwtfqsSY/9lnyJ5Fo4dFTkNPREVSRIeRTGF2YzYyLjI4LjEwMSBzcnRn", + "yKFFo4hEVVJBVElPTkSHkzAwOjAwOjA0LjUwMDAwMDAwMAAfQ7Z1QvW/hFFxCHLngQCjQmiBAACAAAACUwYF//9P3EXpvebZSLeW", + "LNgg2SPu73gyNjQgLSBjb3JlIDE2NSByMzIyMiBiMzU2MDVhIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIw", + "MDMtMjAyNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTAgcmVmPTEgZGVibG9j", + "az0wOjA6MCBhbmFseXNlPTA6MCBtZT1kaWEgc3VibWU9MCBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0wIG1lX3Jh", + "bmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MCA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hy", + "b21hX3FwX29mZnNldD0wIHRocmVhZHM9MSBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0", + "ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTAgd2VpZ2h0cD0wIGtl", + "eWludD0yNTAga2V5aW50X21pbj0xIHNjZW5lY3V0PTAgaW50cmFfcmVmcmVzaD0wIHJjPWNyZiBtYnRyZWU9MCBjcmY9NTEuMCBx", + "Y29tcD0wLjYwIHFwbWluPTAgcXBtYXg9NjkgcXBzdGVwPTQgaXBfcmF0aW89MS40MCBhcT0wAIAAAAAJZYiEOiYoABXAoJyhloID", + "6ABmaXJzdCBlbmdsaXNoIGxpbmWbggPooKChmoMF3ABwcmltZXJhIGxpbmVhIGVzcGFub2xhm4ID6KCdoZeCC7gAc2Vjb25kIGVu", + "Z2xpc2ggbGluZZuCA+igoKGagw2sAHNlZ3VuZGEgbGluZWEgZXNwYW5vbGGbggPoHFO7a/O/hBGUtNq7j7OBALeK94EB8YIDKPCB", + "CbuVs4ID6LeP94EC8YIDKPCCAnSyggPou5WzggXct4/3gQPxggMo8IICkrKCA+i7lbOCC7i3j/eBAvGCAyjwggK0soID6LuVs4IN", + "rLeP94ED8YIDKPCCAtOyggPo", + ] +} diff --git a/docs/architecture.md b/docs/architecture.md index 03aa43e9..f921420a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -181,7 +181,7 @@ Sources/AetherEngine/ │ ├── VideoRoutingPolicy.swift Pure codec-and-field-order dispatch rule: AV1 gated on HW, VP9/VP8/MPEG4/MPEG2/VC1 always SW, interlaced H.264 SW so bwdif can deinterlace (#107, verified against decoded frames on VOD, #232), plus a second-stage gate routing H.264 High 4:2:2/4:4:4/10 + HEVC Rext to SW where VideoToolbox has no HW decoder (#2) │ ├── HardwareVideoDecoder.swift SW path: VideoToolbox HW HEVC / AV1 decoder for sources routed away from AVPlayer │ ├── SoftwareVideoDecoder.swift SW path: libavcodec/dav1d → CVPixelBuffer (NV12 / P010), HDR10+ side data -│ ├── SubtitleDecoder.swift Sidecar URL one-shot decode (text only) +│ ├── SubtitleDecoder.swift Sidecar URL one-shot decode (text only); decodes several streams of one container in a single pass (#266) │ └── VideoDecoderTypes.swift DecodedFrameHandler typealias + VideoDecoderError ├── Demuxer/ │ ├── AVIOProvider.swift Internal seam over a custom-AVIO byte source; AVIOReader and CustomIOReaderBridge both plug into the Demuxer through it, incl. the bounded-seek read deadline and the resolved byte size backing the byte-estimate seek fallback (#112) @@ -253,7 +253,7 @@ Sources/AetherEngine/ │ └── SampleBufferRenderer.swift SW path: AVSampleBufferDisplayLayer + B-frame reorder, HDR10+ attachments; `flush(removingDisplayedImage:)` holds the last frame through a seek (`DisplayFlushOp`, #90) ├── Subtitles/ │ ├── ASSScriptBuilder.swift Reassembles raw ASS event cues + TrackInfo.assHeader into a complete script for whole-file renderers -│ ├── ExternalSubtitleTrack.swift Host-facing descriptor for external subtitle files registered as first-class tracks (synthetic TrackInfo ids, #88) +│ ├── ExternalSubtitleTrack.swift Host-facing descriptor for external subtitle files registered as first-class tracks (synthetic TrackInfo ids, #88; sourceStreamIndex addresses one stream of a container URL, #266) │ ├── Issue100PGSStaleArrival.swift Holdback (`PGSStaleArrivalGate`) for PGS cues arriving behind the playhead: catch-up bursts resolve via their successor's trim instead of flashing open-ended placeholder windows through the overlay (#100) │ ├── MovTextSampleBuilder.swift Stateless tx3g (mov_text) sample builder for the native legible-subtitle injection path (LoadOptions.prepareNativeSubtitles, #55) │ ├── NativeSubtitleCueStore.swift Owns the decoded-cue array behind a native WebVTT subtitle rendition + the overlay tap feed; deduped, filled by the pump tap (embedded) or one whole-file decode (load-declared external, #88) (#55, Sodalite#32) diff --git a/docs/formats.md b/docs/formats.md index afe8176a..dbab0d60 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -94,9 +94,10 @@ Subtitle cues come from one read: EVERY embedded subtitle stream (text and bitma External subtitle files register with the engine and appear in `subtitleTracks` next to the embedded streams, so a host keeps one track list and one selection call (#88): - **Registration.** `LoadOptions.externalSubtitles: [ExternalSubtitleTrack]` declares files at load; `addExternalSubtitleTrack(_:)` registers any time mid-session (returns the created `TrackInfo`). The descriptor carries `url`, optional `name` / `language` / disposition flags, per-track `httpHeaders` (nil forwards the session's), and a `formatHint` for URLs whose path hides the extension. +- **Containers with several subtitle streams.** An external URL can be a container rather than a sidecar (an MKV holding English, English SDH and Spanish). `ExternalSubtitleTrack.sourceStreamIndex` names which stream to decode as an ABSOLUTE `AVStream` index inside that container, so a host registers one track per stream against the same URL; nil decodes the container's first subtitle stream. An index that is out of range or names a non-subtitle stream fails the decode rather than falling back, which would be indistinguishable from leaving it nil. Tracks sharing a URL and headers are filled from a SINGLE pass over the container, so N tracks cost one fetch, not N (#266). Note that `TrackInfo.codec` is still derived from the URL extension, so a container URL reports `subrip`; set `formatHint: "ass"` when the streams are ASS and the host drives a styled renderer. - **Identity.** External `TrackInfo.id`s are synthetic: `AetherEngine.externalSubtitleTrackIDBase` (100 000) + registration ordinal, monotonic per load; load-declared tracks get `base + array index` in order. `TrackInfo.isExternal` distinguishes them from AVStream-indexed embedded tracks. - **Selection.** `selectSubtitleTrack(index:)` and `selectSecondarySubtitleTrack(index:)` accept external ids and route onto the whole-file decode internally; `activeSubtitleTrackIndex` publishes the external id like any other selection. `removeExternalSubtitleTrack(id:)` unregisters (an active selection is cleared). -- **Renditions.** Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. +- **Renditions.** Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank `.vtt`. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. - **Preferences.** `preferredSubtitleLanguages` ranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off. ### Track selection by language preference