Skip to content

fix(gs): correct depth write for all AFAIL alpha-fail modes - #188

Open
smmathews wants to merge 1 commit into
ran-j:mainfrom
smmathews:feature/25-rasterizer-depth
Open

fix(gs): correct depth write for all AFAIL alpha-fail modes#188
smmathews wants to merge 1 commit into
ran-j:mainfrom
smmathews:feature/25-rasterizer-depth

Conversation

@smmathews

@smmathews smmathews commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

fix(gs): correct depth write for all AFAIL alpha-fail modes

Corrected premise

This PR fixes a defect in the alpha-test-fail (AFAIL) truth table that sits next to the depth logic.

The bug

TEST.AFAIL documents four outcomes for a fragment that fails the alpha
test:

AFAIL color write depth write
KEEP (0) no no
FB_ONLY (1) yes no
ZB_ONLY (2) no yes
RGB_ONLY (3) rgb only no

classifyAlphaTest in ps2xRuntime/src/lib/ps2_gs_rasterizer.cpp used a
single writeFramebuffer flag to stand in for both the color and depth
disposition, and writePixel bailed out early only when that flag was
false. Every AFAIL case that survived the early-out then fell through to
an unconditional depth write (gated only by ZMSK). That made every
non-KEEP case except ZB_ONLY write depth on alpha-fail, against the table:
FB_ONLY and RGB_ONLY wrote Z when the table says they should not, and
ZB_ONLY (folded into the same branch as KEEP) wrote neither color nor its
required Z.

The fix

AlphaTestResult gains an independent writeDepth field so each AFAIL
case states its own color and depth disposition, instead of a single
writeFramebuffer flag standing in for both. writePixel gates the color
WriteVram call on writeFramebuffer and the depth WriteVram call on
writeDepth separately. The Z test itself is unchanged: a fragment that
fails the depth test still writes nothing, matching a passing fragment's
depth-test behavior.

A normal alpha-test pass still writes both color and depth. KEEP is
unchanged (it hits the early-out with both flags false, same as before).
FB_ONLY and RGB_ONLY now stop writing depth on alpha-fail, and ZB_ONLY
gains its previously-missing depth-only write.

Test

Three regression tests sit next to the existing AFAIL FB_ONLY and RGB_ONLY
framebuffer pins, each programming ZMSK=0 with a Z page distinct from the
color page and a pre-seeded depth sentinel so the depth path is actually
exercised:

  • AFAIL=ZB_ONLY: forces the alpha test to fail with ZTST=ALWAYS, asserts
    the framebuffer stays untouched while depth is overwritten with the
    drawn fragment's z.
  • AFAIL=FB_ONLY: same setup, asserts the framebuffer is overwritten with
    the drawn color while depth stays at its pre-seeded sentinel.
  • AFAIL=RGB_ONLY: same setup, asserts RGB is written and destination alpha
    preserved while depth stays at its pre-seeded sentinel.

Repro

From the repository root:

cmake -S . -B build -DCMAKE_CXX_FLAGS="-msse4.1 -include cstdint" -DCMAKE_C_FLAGS=-msse4.1
cmake --build build --target ps2x_tests
./build/ps2xTest/ps2x_tests

The GS documents a four-way truth table for what TEST.AFAIL does when
the alpha test fails: KEEP writes neither color nor depth, FB_ONLY
writes color only (no Z), ZB_ONLY writes depth only, and RGB_ONLY
writes RGB while preserving destination alpha (no Z).

The old rasterizer used a single writeFramebuffer flag to stand in for
both the color and depth disposition, and writePixel bailed out early
only when that flag was false. Every case that survived the early-out
then fell through to an unconditional depth write gated only by ZMSK.
That made pass, FB_ONLY, and RGB_ONLY all write depth on alpha-fail,
against the table, while ZB_ONLY was folded into the same branch as
KEEP and wrote neither color nor its required Z.

Give AlphaTestResult an independent writeDepth field so each AFAIL
case states its own color and depth disposition, and gate the two
WriteVram calls in writePixel on writeFramebuffer and writeDepth
separately instead of bailing out on a single flag. Each case now
matches the table: FB_ONLY and RGB_ONLY stop writing Z on alpha-fail,
and ZB_ONLY gains its previously-missing depth-only write. A passing
fragment still writes both. The depth test itself is unchanged; a
fragment that fails it still writes nothing.

Add depth-behavior pins for all three affected AFAIL modes, each with
ZMSK=0, ZTST=ALWAYS, and a Z page distinct from the color page so the
depth path is actually exercised: ZB_ONLY asserts depth is written on
alpha-fail, and FB_ONLY and RGB_ONLY siblings assert depth is left
untouched. The existing FB_ONLY and RGB_ONLY framebuffer tests are
unchanged.
@smmathews
smmathews marked this pull request as ready for review July 27, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant