External subtitle tracks cannot address a container's second subtitle stream

[edde746]

What should the engine do?

An external subtitle URL can be a media container holding several subtitle streams, and a host registers one ExternalSubtitleTrack per stream against that same URL.

decodeFile scans for AVMEDIA_TYPE_SUBTITLE and stops at the first match (SubtitleDecoder.swift:134-143), and ExternalSubtitleTrack carries no index (ExternalSubtitleTrack.swift:9-30). Every such track therefore decodes the first subtitle stream and renders identical cues, and each one refetches the whole container.

Ask: ExternalSubtitleTrack.sourceStreamIndex: Int32?, honoured by SubtitleDecoder.decodeFile.

• Absolute AVStream index within the container, not the Nth subtitle stream.
• nil keeps the current first-subtitle-stream behaviour.

This mirrors the existing audioSourceStreamIndexOverride (HLSVideoEngine.swift:1162-1175) — an absolute stream index validated by stream type via isAudioStream(demuxer:index:) — and the convention that embedded track ids are AVStream indices (RemoteHLSMediaSelection.swift:14). It differs only in reporting an invalid index instead of falling back to auto, since for subtitles a silent fallback is indistinguishable from the current behaviour.

Motivating media or use case

A container carrying several embedded subtitle streams — an MKV with English, English SDH and Spanish ASS tracks — reached through external track registration rather than as the played media. Registering one track per stream yields three selectable tracks that all render the first stream.

Area

Public API surface

Host app / integration

Custom

Would you be willing to open a PR?

Maybe, with guidance

---

Line references are from 6.2.1 (4e281c9).

[superuser404notfound]

Confirmed at the lines given, and implemented in #267 (da52090d), branch feat/issue-266-external-subtitle-stream-index. Thanks for the precise report, the two file references and the pointer at audioSourceStreamIndexOverride as precedent made this a short read.

ExternalSubtitleTrack.sourceStreamIndex: Int32?, honoured by SubtitleDecoder.decodeFile and by both selection paths (primary and the #47 secondary channel). Absolute AVStream index inside the container at url, nil keeps the first-subtitle-stream behaviour, and an index that is out of range or names a non-subtitle stream throws SubtitleDecoderError.streamIndexNotSubtitle rather than falling back.

One naming detail worth flagging, since it is invisible from outside: NativeSubtitleTrackEntry.sourceStreamIndex already exists internally with a different meaning (the stream in the MAIN demuxer that arms the subtitle pump tap, deliberately nil for external entries). The new field indexes a different container, so it is routed through the external registry instead and the two never mix. The public name stays sourceStreamIndex because from a host's side it is the same concept.

The refetch is fixed too, since the index alone would have made it worse. Registering three tracks against one MKV was previously three useless tracks, so nobody paid; with the index they become useful and would have cost three full container reads at load, plus another per styled-ASS selection. decodeFile now also accepts several stream indices and decodes them from a single pass, and load-declared tracks sharing a URL and headers are grouped into one fill job. AVDISCARD_ALL was not an option here: a Matroska demuxer reads every discarded byte anyway, so only a single pass actually saves the bytes. A shared pass fails as a whole, so on failure the targets are retried individually and one bad index cannot blank the container's other tracks.

Two limits to be aware of before this lands on real media:

1. "Report an invalid index" has no host-facing channel today. A sidecar decode failure only reaches EngineLog; the host observes an empty overlay and nothing else. The error is thrown and carries the index, but the engine has no error surface to publish it on. If that matters for your integration, a separate issue for a subtitle-decode error signal would be the right shape.
2. TrackInfo.codec is still derived from the URL extension, so an .mkv URL reports subrip for every track on it. For your English/SDH/Spanish ASS case that means preserveASSMarkup will not engage unless the tracks carry formatHint: "ass". The real codec is only known after the network probe, which is well after registration publishes the track, so deriving it automatically would mean either delaying the track list or mutating it later.

Test coverage runs against a 1.7 KB Matroska fixture embedded in the test target (video at stream 0, English at 1, Spanish at 2), so the absolute-index contract is exercised in CI without a fixture fetch. That layout is deliberate: an ordinal reading of index 1 would land on Spanish, so the two semantics are actually distinguished.

Leaving this open for a check against your real container.

[edde746]

Verified against the real container — #267 does exactly what was asked here.

The index is absolute. My host takes it from the server's per-stream index, and that matches ffprobe's AVStream indices exactly on two real files (subtitle streams at 3-6 and at 3-5).

One pass over [3, 4, 5] of a 3.3 GB Matroska returned 37 / 2011 / 1951 cues — three genuinely distinct streams, where all three previously decoded stream 3. Index 5 came back as the English subrip stream, and the container holds only three subtitle streams, so no ordinal 5 exists: absolute is the only reading that produces it. assHeader stayed per stream, nil on the subrip stream beside two ASS ones.

Both limitations you flagged are non-blocking here: I already pass formatHint from the server's per-stream codec, and an invalid index would be my own bug, so EngineLog is enough — no issue needed.

One follow-up worth a look now that it's in: guard the per-stream PlayRes with the normalised \pos rather than assHeader. Pinning every stream to one PlayRes fails a position assertion but passes a header comparison: comparing assHeader only checks that each stream's own extradata is passed through, and never exercises the per-stream PlayRes state that normalises \pos.

Good to close from my side.

[superuser404notfound]

Verification with those numbers is the useful kind: index 5 on a container holding three subtitle streams is exactly the observation that separates absolute from ordinal, and no fixture can make that argument as cleanly as a real 3.3 GB Matroska does.

The follow-up is in on main as 337f82e2, and the point behind it was right. The suite asserted the header, which is carried per stream whether or not the decode loop shares anything, so it could never have failed for the reason it was watching. The PlayRes is the state a one-pass loop could actually collapse, since it is decode state rather than something read back off the container.

New fixture: a 2.8 KB Matroska with two ASS streams declaring 640x480 and 1920x1080, both carrying the same \pos(320,240). Same tag, different declarations, so the normalized placements can only differ if each stream resolved its own header. Four tests run over it: the shared multi-stream pass, the single-stream path, the fill-job route into the cue stores, and the header property, kept as the weaker statement it is.

Checked by construction rather than by assertion: pinning both streams to 640x480 in StreamDecode fails the three position tests (Spanish off by 0.333 in x and 0.278 in y) while preserved ASS headers stay per stream still passes. That is the gap named here, reproduced and then closed. Full suite green at 1365 tests.

Test-only, so no release: 6.3.0 already behaves this way, the coverage just now says so.

Closing on the verification. Thanks for the report, the two file references in it, and for taking the time to measure the result against real containers afterwards.