ChunkFetchDelegate reserves the whole file's Content-Length on a bodyless HEAD probe: malloc returns NULL and the force-unwrap kills the app on large sources

[dlev02]

What happened?

Opening a large progressive HTTP source crashes the app outright — EXC_BREAKPOINT / "Unexpectedly found nil while unwrapping an Optional value" — before a single frame plays. It reproduces on a 12.4 GB MKV; small files are fine.

The crashing thread is a URLSession delegate queue:

0  Foundation  Swift runtime failure: Unexpectedly found nil while unwrapping an Optional value
1  Foundation  __DataStorage.init(capacity:) + 284
2  Foundation  Data._Representation.reserveCapacity(_:) + 644
3  AetherEngine  specialized ChunkFetchDelegate.urlSession(_:dataTask:didReceive:completionHandler:)  (AVIOReader.swift:2215)
4  CFNetwork   __81-[__NSCFURLSessionDelegateWrapper dataTask:didReceiveResponse:completionHandler:]_block_invoke

x0 is 0x0 in the crashed thread's register state — that is malloc returning NULL, and it's the value being force-unwrapped. x19/x2 hold 0x318ffc000 (13,304,315,904 — 12.39 GiB), consistent with the requested capacity being the source's full Content-Length.

The cause is ChunkFetchDelegate.urlSession(_:dataTask:didReceive:completionHandler:):

self.response = response
if let http = response as? HTTPURLResponse {
    let len = Int(http.expectedContentLength)
    if len > 0 { body.reserveCapacity(len) }
    ...
}

The reservation is whatever the origin declares, with no ceiling. ChunkFetchDelegate is reached through syncRequest(_:budget:), and one of syncRequest's callers is headProbeFileSize() — a HEAD request. A HEAD response carries Content-Length: <entire file> and delivers zero body bytes, so the delegate asks the allocator for the whole movie in order to buffer nothing. On a 6 GB device a 12.4 GiB malloc fails, and __DataStorage.init(capacity:) force-unwraps the NULL.

Two other threads in the same report were parked in headProbeFileSize() → syncRequest → awaitSignal waiting on exactly that response, which lines up: the source needed the HEAD fallback (the range probes hadn't yielded a length), and the response handler blew up on arrival.

headProbeFileSize() is the clearest trigger, but the same line is reachable from ordinary bounded chunk fetches whenever an origin ignores Range and answers 200 with the full length instead of 206 — the reservation is then the whole file again rather than chunkSize.

Worth noting this is a distinct failure from the gradual memory-exhaustion reports (#31, #220, #243): there's no growth curve and no jetsam here. It's one oversized allocation at open time, and it traps rather than throwing, so a host app can't catch or degrade it.

Steps to reproduce

1. Host an MKV large enough that a single malloc of its byte length fails on the target device (12.4 GB reproduced this on a 6 GB iPhone; an Apple TV's 4 GB should bite lower).
2. Serve it from an origin that makes the reader fall back to the HEAD probe — i.e. the bytes=0- range probes don't resolve a length.
3. Open it on the FFmpeg/AVIOReader path.

Crash lands on the URLSession delegate queue as the HEAD response arrives, before playback.

Suggested fix

Cap the reservation, and skip it entirely where no body is coming:

if let http = response as? HTTPURLResponse {
    // Size probes (HEAD, and Range GETs that origins answer with the full
    // length) declare the whole source while delivering little or no body,
    // so the declared length is not a safe reservation.
    if dataTask.currentRequest?.httpMethod != "HEAD" {
        let len = Int(http.expectedContentLength)
        if len > 0 { body.reserveCapacity(min(len, Self.maxBodyReserve)) }
    }
    ...
}

maxBodyReserve at a small multiple of chunkSize would keep the allocation useful for real chunk fetches while making a bogus or whole-file length harmless. Data grows on demand, so the cap costs nothing but a reallocation in the rare case a legitimate body exceeds it.

AetherEngine version or commit SHA

Crash captured on d34e02dd (5.23.7). The line is byte-for-byte identical on 2bfcbd19 (6.0.0) and on 6.1.0, so current main is still affected.

Host app

Custom / my own integration

Platform

iOS

OS version

iOS 26.5.2 (23F84)

Device / chip

iPhone 14 Pro (iPhone15,2), 6 GB RAM

Playback path

FFmpeg / software (custom AVIO)

Source media (for playback bugs)

MKV, ~12.4 GB, over HTTP

[superuser404notfound]

Fixed in bef78fec, released as 6.1.2.

The diagnosis was exact, down to the register reading: NULL out of malloc in x0, the full Content-Length in x19/x2, and the two threads parked in headProbeFileSize() waiting on the very response that killed the process. Thanks for taking the time to symbolicate it and to find the line, this one would have been rough to locate from a crash count.

One thing worth adding, because the surface is wider than origins that reject Range. probeFileSize() runs a staggered ladder: the HEAD fallback is launched 0.75 s after the bytes=0- probe and executes whenever nothing has resolved a size yet. A merely slow origin is enough. So any non-prefetch open (still extraction, one-shot seekable) against a large source could take this, not just Range-hostile ones.

I took the suggested shape but derived the number from the request rather than the response, since the request already knows the exact answer while a multiple of chunkSize would still be a guess:

static func expectedBodyBytes(for request: URLRequest) -> Int? {
    if request.httpMethod?.uppercased() == "HEAD" { return 0 }
    guard let value = request.value(forHTTPHeaderField: "Range") else { return nil }
    return boundedRangeSpan(value)      // nil for bytes=a-, suffix and multi-range forms
}

A HEAD reserves nothing, a bounded chunk fetch reserves at most its own span (clamped against the declared length, so a short tail still reserves only its real size), and an open-ended request falls back to a flat 8 MB ceiling. Nothing on this path issues one of those today.

The second point got the same treatment, and it needed more than a reservation cap: on a Range-ignoring 200 the whole file was reserved and then actually buffered, because the caller's "server ignored Range" rejection only runs after syncRequest returns. The 12 GB allocation was going to be filled before being thrown away. The body is now bounded by the same span and the task cancelled mid-body, with the resulting cancellation not counted as a failed fetch. Against a scripted origin that declares 12.4 GB and streams 64 MB, for a 1 MB chunk request: before, 134 MB pulled over the wire and the read failed outright (each fetch buffering until its budget expired); after, 2 MB written, peak reservation 1 MB, and the kept prefix serves the read.

Two neighbours checked while I was in there, both unaffected: the persistent and streaming reader paths bound their resident window at 48 MB through their own backpressure cap whatever the origin does, and HTTPDiscIOReader never reserves from a declared length and requires working Range support at init.

Leaving this open until you have had a chance to retest 6.1.2 against the 12.4 GB source. Expected: the open completes, and if that origin does ignore Range, one verbose [AVIOReader] response body ran past the requested range line per chunk instead of a crash.