VAAPI "auto" quality sends compression_level 0 to the driver; Intel HD 620 rejects it for HEVC and every stream aborts

[dros74]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

Since #5388, src/platform/linux/vaapi.cpp sets ctx->compression_level = target_quality whenever the driver reports a quality range, and with vaapi_quality = auto (the default) target_quality is 0 (lines 317 to 334 at v2026.921.13701). libavcodec's vaapi_encode.c treats any compression_level >= 0 as set and passes it as VAEncMiscParameterBufferQualityLevel, so the driver receives quality level 0.

On my Intel HD Graphics 620 (Kaby Lake) with intel-media-driver 26.2.4, the driver accepts HEVC quality levels 1, 4 and 7 only. Level 0 fails at vaEndPicture with Failed to end picture encode issue: 24 (internal encoding error). Sunshine then aborts in the session::video thread (SIGABRT, stack below), systemd restarts it, Moonlight reports the stream ending, and the next connection does the same. Five core dumps in two minutes this morning. The encoder validation at startup passes and logs [VAAPI] Quality level set to 0 (fastest level: 7), so nothing looks wrong until a client connects.

The previous build I had (2026.516.143833, before #5388) did not set a quality level and streamed fine on the same machine, same driver.

Reproduction outside Sunshine, same GPU and driver, system FFmpeg 9.0.2:

ffmpeg -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va \
  -f lavfi -i testsrc2=size=1920x1080:rate=30 -t 1 -vf format=nv12,hwupload \
  -c:v hevc_vaapi -compression_level 0 -b:v 23M -maxrate 23M -bufsize 766k -f null -

Results by -compression_level value:

hevc_vaapi  0        fails: Failed to end picture encode issue: 24
hevc_vaapi  1, 4, 7  ok
hevc_vaapi  2, 3, 5, 6  fail (same error)
hevc_vaapi  unset    ok (libavcodec: "No quality level set; using default")
h264_vaapi  0 to 7   ok

Workaround that fixed it for me: vaapi_quality = speed (level 7). quality (level 1) also works. balanced computes 7 / 2 = 3, which this driver rejects as well, so two of the four documented choices crash on this hardware.

Expected Behavior

vaapi_quality = auto should leave ctx->compression_level at its libavcodec default (-1) so the driver picks its own default quality, which is what 2026.516 did. Something like: only assign compression_level in the speed, balanced and quality cases. For balanced, (1 + quality_attr.value) / 2 gives 4 on Intel, which is the driver's middle preset, instead of 3.

Additional Context

Not the same as #5471: that one is a null dereference (segfault at 28) on 2026.516, before any frame is encoded. This one is an abort after the driver rejects the first frame, and only exists since #5388.

I ran the CachyOS package of v2026.921.13701 (log reports commit 019b1ba). I did not run the v2026.922.203725 binary; I checked that vaapi.cpp is unchanged there and on master apart from a designated initializer.

Moonlight client was the iOS app over Tailscale, requesting HEVC 8-bit at 30 fps. sunshine.conf was empty when it crashed.

Host Operating System

Linux

Operating System Version

CachyOS (Arch based), kernel 7.2.5-cachyos, niri 26.04 (Wayland), greetd

Architecture

amd64/x86_64

Package

other (not listed)

CachyOS repository package sunshine 2026.921.13701-1

GPU Type

Intel

GPU Model

Intel HD Graphics 620 (Kaby Lake-U GT2), Core i5-7200U

GPU Driver/Mesa Version

intel-media-driver (iHD) 26.2.4, libva 2.24.1, Mesa 26.2.3

Capture Method

wlroots (FreeBSD/Linux)

Config

# empty at the time of the crash; workaround now in place:
vaapi_quality = speed

Apps

Default Desktop entry, no commands.

Log output

[2026-09-24 09:15:35.175]: Info: CLIENT CONNECTED
[2026-09-24 09:15:35.397]: Info: Screencasting with Wayland's protocol
[2026-09-24 09:15:35.501]: Info: Creating encoder [hevc_vaapi]
[2026-09-24 09:15:35.501]: Info: Color depth: 8-bit
[2026-09-24 09:15:35.530]: Info: vaapi vendor: Intel iHD driver for Intel(R) Gen Graphics - 26.2.4 ()
[2026-09-24 09:15:35.530]: Info: Streaming bitrate is 22988000
[2026-09-24 09:15:35.530]: Info: Using normal encoding mode
[2026-09-24 09:15:35.530]: Info: [VAAPI] Quality level set to 0 (fastest level: 7)
[2026-09-24 09:15:35.530]: Info: [VAAPI] Using VBR rate control with single frame VBV size (whitelist override)
[2026-09-24 09:15:35.530]: Warning: [hevc_vaapi @ 0x7ffbb406ec40] Driver does not advertise encoder features, using guessed defaults.
[2026-09-24 09:15:35.530]: Warning: [hevc_vaapi @ 0x7ffbb406ec40] Driver does not advertise encoder block size, using guessed defaults.
[2026-09-24 09:15:35.607]: Info: Minimum FPS target set to ~15fps (66.6667ms)
[2026-09-24 09:15:35.645]: Error: [hevc_vaapi @ 0x7ffbb406ec40] Failed to end picture encode issue: 24 (internal encoding error).

systemd / coredumpctl:

systemd-coredump: Process 2462624 (sunshine) of user 1000 dumped core.
app-dev.lizardbyte.app.Sunshine.service: Main process exited, code=dumped, status=6/ABRT
Stack trace of thread 2541429 (session::video):
#0  pthread_kill (libc.so.6 + 0xaeceb)
#1  raise (libc.so.6 + 0x44d48)
#2  abort (libc.so.6 + 0x257c1)
#3  n/a (sunshine + 0x478cc4)
#4  n/a (sunshine + 0x4791a3)
#5  n/a (sunshine + 0x3fec61)
#6  n/a (sunshine + 0x3fee97)
#7  n/a (sunshine + 0x2629d0)
#8  n/a (sunshine + 0x25469c)

After vaapi_quality = speed and a restart: [VAAPI] Quality level set to 7 (fastest level: 7), stream and input work.

Online logs

No response

[tajemniktv]

Managed to repro on Kaby Lake

[psyke83]

Please try #5797 to fix the two issues you identified.

We need to ensure that vaCreateConfig properly guards against invalid levels (specifically 0, as balanced will now resolve to 4 and should work).