feat!: default add-on release workflows to Python 3.13 with 3.9 override (ADDON-89519)

[mbruzda-splunk]

Summary

• Run package build, unit tests, pre-commit, and WFE test tooling on Python 3.13 by default. Callers can set python-version: "3.9" to run all of those jobs and WFE test dependencies on Python 3.9.
• Remove the redundant parallel Python 3.13 unit-test and build jobs; the primary build remains the package artifact consumed downstream.
• Update this repository's CI, Splunkbase publishing, and documentation workflows to Python 3.13.
• Handle test summary artifacts whether actions/download-artifact extracts them at the workspace root or inside an artifact directory; remove an invalid SPL2 matrix reference.
• Keep the python-version input and the deprecated test-python-version alias. python-version takes precedence; if neither is set, the workflow uses 3.13.

Coordinated downstream PRs and MRs

• Repository template migration PR #936 (dependent draft; preserves Python 3.9 overrides and adds release/canary gates for 77 downstream repositories)
• WFE test runner PR #52
• Kubernetes manifests PR #149
• Argo templates PR #45
• MSCS staging validation PR #1581
• AWS staging validation PR #1767
• CyberArk replacement staging validation PR #374
• Google Workspace staging validation PR #722 (earlier diagnostic runs)
• TA test image MR !249 (optional newer image; the staging TA test path can use the published 2.0.1-python3.13 tag)
• SPL2 test image MR !42
• SPL2 framework NumPy fix PR #63

ECR has both workflow-engine-ta-test-base:2.0.1-python3.9 and workflow-engine-ta-test-base:2.0.1-python3.13. Argo templates PR #45 selects the tag from the python-version parameter and accepts both versions. The published 3.13 container runs Python 3.13.5 and has Poetry 2.1.4 at runtime; WFE pytest scripts do not invoke Poetry. SPL2 is a separate image path: for a 3.9 caller, PR #45 pins its prior Python 3.10 image; for the 3.13 default, the SPL2 framework fix must release, then its image can build and refresh the ECR latest tag. Merge only after the remaining integration checks are resolved.

Review follow-up (2026-10-07)

Current workflow head: 2d9605a3152ddc065a8cfb147d09f6b80b7c9b50.

• Resolved conflicts with current develop (e884eeb), retaining the new PR-scoped TA Validator exception handling. Signed merge commit: d98c8db.
• Addressed the six review comments in signed commits 695cef8 and 2d9605a: documented the wider Python override scope and selected-version artifact contract, corrected conditional unit-test documentation, cross-referenced the two required fallback expressions, rejected unsupported selected versions before pre-commit/build/unit/WFE jobs, and scoped/sorted all seven WFE report aggregations.
• Local validation: 79 tests passed, 84.67% coverage, all pre-commit hooks and actionlint passed. Shell fixtures exercise supported/invalid values and all seven report jobs with zero/one/multiple, flat/nested, wrong-prefix and ordering scenarios.
• Current-head repository CI passed, including compliance, secret scanning, pre-commit/unit tests, Semgrep and the CI gate. Publication jobs correctly skipped for the PR event. All six addressed review threads have replies and are resolved.
• Isolated full fixture PR #388 targets main without suite-selection labels. Its initial unsupported Python 3.10 run completed as expected: rejected the value in the first setup step; pre-commit, unit, build, setup and every WFE submission/report job skipped. The validation failure is expected for this negative scenario; independent FOSSA vulnerability/license checks also failed. An unchanged separate fixture docs workflow failed as well. No WFE jobs were submitted. Caller head 10c30cafa13f07968714ad3a81efc9c4eaa485df, baseline 2290aa46709754336f43761075f2ac05f48edb91. Full default-Python-3.13 staging run 37633314743 completed on caller 4300c9e788b2e36ff75611d980484169f123c267, targeting fixture main without suite-selection labels. Unit, pre-commit, build, setup and all five btool checks passed. All seven modinput variants and five UI variants produced clean JUnit results. Knowledge had exactly the three deliberate sample failures per Splunk version. Modinput, knowledge and UI summary report jobs all passed, aggregating 7, 5 and 5 artifacts respectively. The aggregate run is red because of the intentional knowledge failures, a FOSSA CLI download HTTP 500, and GitHub check-update HTTP 404s in one modinput/two UI jobs whose test results passed. See the detailed current-head fixture section below. No Terraform files changed. The completed MSCS/AWS/CyberArk runs below remain evidence for their recorded earlier workflow SHAs, not this new head.

Current-head full fixture validation (2026-10-07)

• Workflow: 2d9605a3152ddc065a8cfb147d09f6b80b7c9b50.
• Fixture PR #388, baseline main at 2290aa46709754336f43761075f2ac05f48edb91, caller 4300c9e788b2e36ff75611d980484169f123c267. PR synchronize event, base main, no suite-selection labels; default Python 3.13 with no override. Shared main/develop callers were never modified.
• Full run 37633314743 exercised Splunk 9.4.16, 10.0.11, 10.2.8, 10.4.4 and 10.6.0, including both server-Python modes on the two legacy versions. All 22 submitted Argo workflows carried python-version=3.13 and finished. TA pytest containers used workflow-engine-ta-test-base:2.0.1-python3.13; logs confirmed Python 3.13.5. Btool retains its existing infrastructure base image workflow-engine-base:4.1.0.
• Coordinated Argo templates: exact commit 17c4730cff2b1842b5cd2b6e718c81f87392de9a; manifests branch feat/ADDON-89519-python-313 at verified commit 573e7640fbfe87b430ecbec18e4a2e98667b3b7b.

Path	Verified result
Python validation / pre-commit / build / setup	Passed
Unit	198 passed, 0 failures/errors/skips; selected-version artifact test-results-unit-python_3.13
Btool	All 5 checks passed
Modinput	196 passed across 7 variants, 0 failures/errors/skips; 6 GitHub jobs green and 1 red only because test-reporter check update returned HTTP 404
UI	40 passed, 5 skipped across 5 variants, 0 failures/errors; 3 GitHub jobs green and 2 red only because test-reporter check update returned HTTP 404
Knowledge	3,655 tests: 3,376 passed, 15 failed, 264 skipped/expected failures, 0 errors. Every version has the same 3 deliberate fixture failures
Modinput / knowledge / UI summary jobs	All passed, aggregating 7 / 5 / 5 artifacts with the new sorted, scoped aggregation
UCC modinput / scripted-input / upgrade / SPL2	Correctly skipped because the fixture lacks those suites; all seven summary shell variants are covered by the repository tests

The 15 knowledge failures are explained by fixture commit 2de6a0e ("break tests for testing purposes"): expected Boxax instead of Box, expected DudmmyUser02 instead of DummyUser02, and mismatched object_id after changing the source ID. Those sample changes and test assertions were retained; no collection errors or additional failed knowledge cases occurred. The run is not fully green. FOSSA failed downloading its unchanged CLI from GitHub with HTTP 500; the three red UI/modinput jobs passed all tests and failed only on the GitHub Checks API HTTP 404. A separate unchanged fixture docs caller at @feat/docs-reusable-workflow failed after printing "Docs validation success" on Python 3.12.14; it does not reference the migrated workflow SHA. The other docs caller at @v5.2 passed.

A btool Argo internal output-resolution warning refers to the skipped test-addon step; its output expression is byte-for-byte unchanged from the saved staging baseline. All btool containers and GitHub btool checks passed.

Cleanup verified: original revisions and full sync policies on all four Argo applications were restored, and ta-workflow / run-tests Python defaults returned to 3.7, exactly as before the run. No active workflows remain. argo-workflow-templates retains its pre-existing OutOfSync status with last operation Succeeded; the other three applications are Synced. No Terraform files changed. The temporary fixture PR is closed after evidence collection; shared fixture branches remain untouched. PR #540 remains a draft pending normal review and coordinated release gates.

Validation

• actionlint passed for all workflows.
• Merged current develop into the feature branch to resolve the PR conflict (commit 2adb89f), preserving Python 3.13 and the summary fix alongside the new repository validation workflow. Local poetry run pytest passed (11 tests, 96.67% coverage), and all pre-commit hooks passed.
• Parsed all workflow YAML files and verified git diff --check.
• Repository CI passed.
• MSCS staging run 36713011612, attempt 3 exercised this reusable workflow through MSCS PR #1581, Argo templates PR chore(deps): bump actions/setup-python from 2 to 3.1.1 #45, and manifests PR chore: update CODEOWNERS #149. Python 3.13 unit tests, pre-commit, package build, staging btool, staging knowledge tests, and the knowledge report job passed. The Argo workflows used python-version=3.13 and the manifests branch from PR chore: update CODEOWNERS #149; the live test pod used workflow-engine-ta-test-base:2.0.1-python3.13 (Python 3.13.5, Poetry 2.1.4). Knowledge results: 3,358 passed, 2 skipped, 199 xfailed, 6 xpassed, with no failed tests. This run confirms the report artifact layout fix that failed in attempt 2. The WFE runner PR chore(deps): bump docker/login-action from 1.12.0 to 2.0.0 #52 and SPL2 changes remain separate validation work.
• MSCS Python 3.9 staging run 36989518728 exercised the restored override against reusable workflow commit 727ee2b and the updated Argo templates. Unit tests, pre-commit, package build, staging btool, staging knowledge, and the knowledge report passed. The live Argo workflows received python-version=3.9; the knowledge pod used workflow-engine-ta-test-base:2.0.1-python3.9. Knowledge results: 3,358 passed, 2 skipped, 199 xfailed, 6 xpassed, with no failed tests.
• MSCS default Python 3.13 staging run 36991339027 exercised the same reusable workflow commit with no version override. Unit tests, pre-commit, package build, staging btool, staging knowledge, and the knowledge report passed. Live Argo workflows received python-version=3.13; the knowledge pod used workflow-engine-ta-test-base:2.0.1-python3.13. The JUnit artifact records 3,565 tests, 0 errors, 0 failures, 201 skipped or expected failures.
• Staging Argo repository authentication was repaired, the PR chore(deps): bump actions/setup-python from 2 to 3.1.1 #45 templates were deployed for this test, and staging revisions and auto-sync settings were restored afterward. No Terraform files changed.
• Open gates: both MSCS compatibility runs are red because cloud AppInspect reports check_all_python_files_are_well_formed and FOSSA reports active vulnerability issues (4 in the 3.9 run). The earlier Python 3.13 attempt also failed cloud AppInspect on bundled dependency syntax checks, while the unmodified develop baseline passed it. These gates need investigation before merging the migration. SPL2 was skipped by this MSCS workflow and needs separate validation.

Additional add-on staging validation (2026-10-05)

Both callers pin reusable workflow commit 727ee2be0ed0b59b05318daa12834837002dba73 and use Argo templates commit 17c4730cff2b1842b5cd2b6e718c81f87392de9a plus manifests commit 573e7640fbfe87b430ecbec18e4a2e98667b3b7b. These are pull_request synchronize events against develop, with execute_knowledge enabled on Splunk 10.6.0 (86587d4e3b27). Unit tests, pre-commit, build, AppInspect, staging setup, btool, knowledge and the knowledge report are selected; modinput, UCC modinput, UI, scripted-input, upgrade and SPL2 suites are skipped. All four staging knowledge pods used the matching published workflow-engine-ta-test-base:2.0.1-python3.9 or 2.0.1-python3.13 image, and live Argo parameters matched the caller scenario. Add-on dependencies and Terraform files are unchanged.

Add-on	Python scenario	Caller head	GitHub run	Result
AWS	Explicit 3.9 override	f4d92d7	37310077494	379 unit tests, pre-commit, build, cloud AppInspect, staging setup, btool, knowledge and knowledge report passed. Knowledge JUnit: 8,744 tests, 0 failures/errors, 1,069 skipped/expected failures. Legacy splunktafunctionaltests policy and FOSSA vulnerability gates (2 active issues) failed. Overall failure.
Google Workspace	Explicit 3.9 override	87298dc	37309833349	141 unit tests, pre-commit, build, cloud AppInspect, staging btool and the knowledge report passed. Knowledge JUnit: 2,650 tests, 1 failure, 0 errors, 123 skipped/expected failures. test_splunk_internal_errors found SPL2 language-server socket/health-check errors in Splunk. FOSSA: 2 vulnerability and 3 licensing issues. Overall failure.
AWS	Default 3.13, no override	0a28efa	37314074104	379 unit tests, pre-commit, build, cloud AppInspect, staging setup, btool, knowledge and knowledge report passed. Knowledge JUnit: 8,744 tests, 0 failures/errors, 1,069 skipped/expected failures. Legacy splunktafunctionaltests policy and FOSSA vulnerability gates (2 active issues) failed. Overall failure.
Google Workspace	Default 3.13, no override	8ecda53	37311999107	141 unit tests, pre-commit, build, cloud AppInspect, staging setup, btool and knowledge report passed. Knowledge collection failed with 8 re.PatternError errors in the pinned pytest-splunk-addon==5.4.0; no knowledge tests executed. FOSSA license and vulnerability gates also failed. Overall failure.

Staging cleanup completed after all eight Argo btool/knowledge workflows finished: original main/main-1.35 revisions and auto-sync/self-heal policies were restored, and live workflow Python defaults returned to 3.7. The workflow-template application retains its pre-existing OutOfSync status with its last operation Succeeded. No Terraform files changed. Both add-on validation PRs remain unmerged drafts for review and follow-up.

The framework Python 3.13 fix is already merged in pytest-splunk-addon PR #928. AWS pins framework 5.5.0, whose released source already contains the source-key regex fix.

Google Workspace follow-up: Google Workspace pins pytest-splunk-addon==5.4.0 in dev_deps/requirements_dev.txt. Its props parser uses (?:(?i)in\s+(\w+))\s*$, which Python 3.13 rejects because the global inline flag is inside a group. Released framework v5.5.0 already moves the flag to the start. Google needs the framework updated to 5.5.0 or a newer compatible release, then both scenarios rerun before knowledge compatibility can be claimed.

Replacement add-on validation: CyberArk

Selected CyberArk from the repository dashboard to replace Google Workspace in the validation set. The current add-on supports >=3.9,<3.14 and already pins pytest-splunk-addon==7.0.4. Its recent full baseline passed unit tests on both versions and knowledge tests across four Splunk versions.

Validation PR #374 starts from main commit 039bf960a237730702cd62e01ab4bf10f1497a28 because the repository has no develop branch. Its base is the unchanged validation branch test/ADDON-89519-staging-baseline at that same commit. PR synchronize events use use_labels and execute_knowledge; the selected checks are unit, pre-commit, build, AppInspect, staging setup, btool, knowledge and the knowledge report. The validation base selects only Splunk 10.6.0 for both Python scenarios. The reusable workflow, Argo and manifests commits are the same as the MSCS/AWS tests above. No add-on dependency or Terraform changes.

Python scenario	Caller head	GitHub run	Result
Initial 3.9 attempt, base main	4199538	37320835187	Cancelled: five-version matrix encountered staging disk pressure and delayed scale-up. 25 unit tests, build, pre-commit, AppInspect and FOSSA gates passed.
Explicit 3.9 override, single-version retry	87322ef	37323208197	Unit 25 passed; pre-commit, build, FOSSA, AppInspect, setup, btool and report passed. Knowledge: 993 tests, 1 failure, 0 errors, 42 skip/xfail. Overall failed solely in knowledge internal-log check.
Default 3.13, no override	6ae152f	37324931280	Unit 25 passed; pre-commit, build, FOSSA, AppInspect, setup, btool and report passed. Knowledge: 993 tests, 1 failure, 0 errors, 42 skip/xfail. Overall failed solely in knowledge internal-log check.

Knowledge logs confirm runtime Python 3.9.23 for the override and Python 3.13.5 for the default, using 2.0.1-python3.9 and 2.0.1-python3.13 images respectively. The two single-version runs intentionally skip modinput, UI, scripted-input, upgrade, UCC modinput and SPL2 suites; AppInspect API cloud is also skipped by branch routing. CLI cloud AppInspect runs and passes.

Both CyberArk runs reproduce the same sole test_splunk_internal_errors failure: Splunk 10.6 SPL2 language-server missing sockets, orchestrator health checks and supervisor errors. All other 950 knowledge cases pass in each run, with 42 skip/xfail and no collection errors. Reproduction on both Python versions points to the Splunk/staging runtime rather than a Python 3.13-only regression; the runs are not fully green. No new ignore rules were added.

Staging restored: all four Argo application revisions and sync policies exactly match the pre-test backup. Workflow templates again use their original Python 3.7 defaults. No active test workflows remain. argo-workflow-templates retains its pre-existing OutOfSync state with last operation Succeeded; the other applications are Synced.

Ticket: https://splunk.atlassian.net/browse/ADDON-89519

[mkolasinski-splunk]

Automated deep review (5 independent lenses: Reliability, K8s/Infra, Security, Maintainability, Shell/Bash — all on Claude, no cross-model run available in this environment).

No high-risk findings. 4 medium-risk items all point at the same underlying theme: consolidating python-version/test-python-version into one PYTHON_VERSION fallback gives that single value a wider effective blast radius than before (which test images run, which artifacts get produced, and how far a malformed value could propagate), plus a minor duplicated-expression sync-drift risk. 2 suggestions: a README accuracy regression for python-version: "3.9" overrides, and minor notes on the new find-based summary aggregation.

Security lens: no findings. Job-dependency-graph cleanup from removing run-unit-tests-py313/build-py313 was independently verified clean by two reviewers (no dangling needs:/if: references). The removed vendor-version matrix reference in the SPL2 job was separately confirmed valid (that job's matrix genuinely has no vendor-version field).

All non-blocking / advisory — posting as COMMENT, not requesting changes.

[mkolasinski-splunk]

@codex

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-07T16:50:37.483030Z	2d9605a	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.