Failed build: MaxText Package Tests

[github-actions]

GitHub Actions workflow MaxText Package Tests #13487 failed.

Event: schedule
Branch: main
Commit: 4110d139cc2a8a3d4fab53cd5568028a7897568f

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13524 failed.

Event: schedule
Branch: main
Commit: d11185b6062096417c67b6a45e0881dfe6579222

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13593 failed.

Event: schedule
Branch: main
Commit: cbafb8f7579a904252b250bd247641a0ceee2d22

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13699 failed.

Event: schedule
Branch: main
Commit: fb2a3f679604388d8ca4e9490bfab9f2d0d7c304

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13798 failed.

Event: schedule
Branch: main
Commit: f80650dfa844210f72867ebd15da5650d84ae319

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13826 failed.

Event: schedule
Branch: main
Commit: be4fd71c8f114bfcbba98572b21cd53f06fa19c2

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13903 failed.

Event: schedule
Branch: main
Commit: 8bc2012e8db4efd0f6683505755e65e03559896f

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13906 failed.

Event: schedule
Branch: main
Commit: 8bc2012e8db4efd0f6683505755e65e03559896f

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13911 failed.

Event: schedule
Branch: main
Commit: 493fba68b3fd62c6159ba39634ca1e60465ca3f1

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #13949 failed.

Event: schedule
Branch: main
Commit: 73293704ef336eea91c780ff73f4864213de5674

^{Created by jayqi/failed-build-issue-action}

[github-actions]

🤖 CI Failure Investigation Report

I have analyzed the recent test failures in the CI pipeline and identified a regression likely introduced in PR #3912 ("deprecate-aqt-phase1"):

🔍 What Failed

• Job/Matrix: MaxText Package Tests (Scheduled/Main)
• Failing Test: tests/unit/quantizations_test.py and integration tests using fp8_full.
• Error: ValueError: Invalid value configured for quantization fp8_full.

🪵 Error Details & Stack Trace

  File "src/maxtext/layers/quantizations.py", line 612, in _get_quant_config
    raise ValueError(f"Invalid value configured for quantization {config.quantization}.")
ValueError: Invalid value configured for quantization fp8_full.

💡 Root Cause Analysis & Context

Confidence: High (for this specific error)

The failure was introduced in PR #3912, which aimed to deprecate AQT quantization. While fp8_full was added to the QuantizationType enum in src/maxtext/configs/types.py, the corresponding logic in src/maxtext/layers/quantizations.py was not updated. Specifically:

1. _get_quant_config is missing fp8_full: When use_qwix_quantization is False (the default), configure_quantization calls _get_quant_config. This function lacks a branch for "fp8_full", causing it to raise a ValueError.
2. Broken Legacy Support: The removal of aqt_fp8 and aqt_fp8_full from the enum causes ValidationError in scripts that still use these tags (e.g., src/maxtext/inference/gpu/microbenchmark_llama2-70b_h100-8.sh), even though the underlying logic still exists in quantizations.py.

Note: While the "MaxText Package Tests" have been failing since late May (likely due to other causes such as HLO graph regressions or environment flakes), this regression will prevent the tests from passing even if those other issues are resolved.

🛠️ Recommended Fix

Update src/maxtext/layers/quantizations.py to handle fp8_full in the _get_quant_config function:

<<<<
  if config.quantization == "aqt_fp8_full":
    return _get_aqt_fp8_default_config(config)
====
  if config.quantization == "aqt_fp8_full" or config.quantization == "fp8_full":
    return _get_aqt_fp8_default_config(config)
>>>>

Additionally, consider restoring aqt_fp8 and aqt_fp8_full to the QuantizationType enum in src/maxtext/configs/types.py if backward compatibility is required during this phase of deprecation.

[github-actions]

GitHub Actions workflow MaxText Package Tests #13992 failed.

Event: schedule
Branch: main
Commit: 0747df9c771f564fc3f319ad788eb2e67882594c

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #14020 failed.

Event: schedule
Branch: main
Commit: 575b31067a938c21af741057a184e8e78ceb36be

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #14128 failed.

Event: schedule
Branch: main
Commit: d8763efa5bbc2d24b15fc5c856f885698f89e01e

^{Created by jayqi/failed-build-issue-action}

[github-actions]

GitHub Actions workflow MaxText Package Tests #14135 failed.

Event: schedule
Branch: main
Commit: b74794138a28073538fdc390315c34de4a0e5a74

^{Created by jayqi/failed-build-issue-action}