feat(tls): add corporate CA bundles for package HTTPS

[Josh Bazar (TameTheGame)]

Description

APM_EXTRA_CA_BUNDLE adds corporate PEM certificates to APM's own package-management HTTPS while retaining its normal default trust roots. This lets a private registry or HTTPS proxy work without replacing the complete Requests trust set or changing the machine trust store.

Trust selection remains in core/tls_trust.py:

• REQUESTS_CA_BUNDLE takes precedence over CURL_CA_BUNDLE; either retains replacement semantics and suppresses OS/additive injection.
• APM_DISABLE_TRUSTSTORE disables OS/additive trust without removing an explicit replacement bundle.
• Otherwise, a selected extra bundle augments OS trust. If truststore is unavailable or injection fails, Requests retains certifi roots plus the extra certificates, and stdlib metadata HTTPS retains its existing default roots plus the extra certificates.
• Unset or blank configuration preserves existing behavior. Invalid selected input fails before command execution with recovery guidance. Certificate and hostname verification stay enabled.

The bundle is validated once in memory. Failed publication restores the previous TLS state. The fallback stays inside the APM process; it does not create certificate files or derive environment variables for children. Hardened package-lifecycle Sessions retain explicit Requests/curl replacement settings while keeping trust_env=False.

Issue and approved scope

Fixes #2034 within the approved package-management scope, recorded on October 2. Review contact: Daniel Meppiel (@danielmeppiel).

This revision removes the earlier execution-runtime expansion: Python/Node propagation, snapshot directories and ownership markers, nested-shell machinery, and managed-runtime refresh. The child bootstrap, script runner, and llm runtime now match upstream. Existing OS-trust bootstrap behavior is preserved. APM_EXTRA_CA_DIR, native Git/Rust TLS configuration, machine trust-store changes, and disabling verification remain outside this contribution.

TLS troubleshooting, environment-variable reference, install diagnostics, enterprise guidance, and packaged apm-usage guidance describe the narrowed behavior and precedence. The changelog entry is under Unreleased; released history is preserved.

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass (the affected suite was run; full hosted CI is separate)
• Added tests for new functionality

Windows / Python 3.12.13:

• Affected TLS, runtime, lifecycle, CLI, and documentation suite: 363 passed, no skips or deselections, in 76.77 seconds. All three symlink tests ran with elevation. The two warnings come from reader threads in the unchanged lifecycle timeout test.
• Final focused acceptance and scope checks: 28 passed. Repository test-quality contracts: 64 passed; assertion-quality and exact-duplicate ratchets passed.
• Full required lint contract passed: Ruff, formatting (1,911 Python files), duplication, architecture/auth boundaries, YAML I/O, file-length and portable-path guards. Whitespace checks passed.
• Documentation built 125 pages, checked 1,071 relative links, and matched all 34 public commands to their rendered reference pages.

The real source CLI installs an actual package from a private-CA loopback HTTPS registry. Ten fresh-project cases cover both normal OS trust and forced truststore unavailability: an independent default-root control, rejection without the extra CA, private-CA success, default-root retention with the extra CA, and rejection when an explicit replacement omits the default root. Successful installs verify metadata/download requests, package bytes, lockfile, and deployed instructions. The fixture uses synthetic roots and test-process certifi seeding, with no machine trust edits or transport/resolver mocks.

Additional real TLS checks cover Requests and stdlib fallback after import/publication failure, default-root retention, rejection of unrelated roots and wrong hostnames, startup failure for invalid input, and real apm run children receiving no derived additive trust. Existing runtime/bootstrap, frozen-hook, lifecycle, CLI, and precedence regressions are included.

This is hermetic compatibility evidence, not a claim of validation in a particular enterprise deployment. The branch includes upstream main at 18c4c43c. Hosted checks and maintainer review remain separate gates.

Spec conformance (OpenAPM v0.1)

• N/A -- this changes opt-in transport trust configuration, not the OpenAPM package format, resolution, policy, or normative lifecycle semantics.

[Josh Bazar (TameTheGame)]

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Adds opt-in additive enterprise CA support via APM_EXTRA_CA_BUNDLE, ensuring APM can trust corporate/private roots in addition to its normal trust (OS truststore when available, otherwise certifi), and propagates stable, validated CA snapshots to child processes (Python Requests children and Node children) without TOCTOU on operator-controlled files.

Changes:

• Introduces APM_EXTRA_CA_BUNDLE with bounded, certificate-only PEM validation; transactional publication of OS-plus-extra TLS context and robust fallback behavior.
• Implements per-process CA snapshotting under ~/.apm/tls/ and derives child env mappings (REQUESTS_CA_BUNDLE merged snapshot; NODE_EXTRA_CA_CERTS extra-only snapshot) with ownership markers for nested runs.
• Updates CLI early-failure handling, runtime/script spawn seams, tests, and docs/changelog to reflect the new precedence and scope boundaries.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/apm_cli/core/tls_trust.py	Core owner for additive CA validation, transactional truststore publication, stable snapshots, child env derivation, and managed-venv bootstrap refresh.
src/apm_cli/core/_child_tls/_apm_tls_bootstrap.py	Managed Python bootstrap updated to support OS-plus-extra injection silently and transactionally in child interpreters.
src/apm_cli/cli.py	Converts early TLS configuration errors into a single Click failure before command callbacks execute.
src/apm_cli/core/script_runner.py	Ensures apm run spawn paths apply build_child_tls_env for both shell and runtime-exec boundaries.
src/apm_cli/core/script_executors.py	Ensures hardened Sessions (trust_env=False) still honor explicit CA replacement variables via the canonical helper.
src/apm_cli/runtime/llm_runtime.py	Ensures managed llm runtime invocations use the canonical child TLS environment (and bootstrap refresh).
src/apm_cli/install/validation.py	Updates TLS failure guidance to recommend additive trust (APM_EXTRA_CA_BUNDLE) before replacement (REQUESTS_CA_BUNDLE).
CHANGELOG.md	Adds an Unreleased entry documenting the new additive enterprise CA behavior and Node non-overwrite semantics.
docs/src/content/docs/troubleshooting/ssl-issues.md	Documents precedence, runtime coverage, failure behavior, and configuration recipes for additive trust.
docs/src/content/docs/reference/environment-variables.md	Documents APM_EXTRA_CA_BUNDLE, NODE_EXTRA_CA_CERTS, and explicit resolution order/scope.
docs/src/content/docs/enterprise/security.md	Updates enterprise security model with additive trust mechanics, snapshots, and precedence boundaries.
docs/src/content/docs/enterprise/registry-proxy.md	Updates proxy troubleshooting to prefer additive trust while retaining public roots.
docs/src/content/docs/troubleshooting/common-errors.md	Aligns common TLS error recovery guidance with additive trust.
docs/src/content/docs/troubleshooting/install-failures.md	Aligns install TLS troubleshooting with additive trust and precedence guidance.
packages/apm-guide/.apm/skills/apm-usage/troubleshooting.md	Keeps packaged troubleshooting guidance in sync with additive trust behavior.
tests/unit/core/test_tls_trust.py	Adds unit coverage for additive validation, rollback, snapshotting, ownership markers, and managed bootstrap refresh.
tests/unit/core/test_script_runner_execution.py	Adds unit coverage to lock TLS child env application at both runtime and shell spawn seams.
tests/unit/test_llm_runtime.py	Updates expectations to ensure managed runtime spawns request the llm-scoped child TLS environment.
tests/unit/test_lifecycle_executor_paths.py	Adds tests proving hardened Sessions honor explicit CA bundle settings even with trust_env=False.
tests/unit/test_tls_docs_scope.py	Updates doc drift guards to enforce runtime scope wording and additive-variable documentation.
tests/integration/test_tls_custom_ca.py	Adds integration coverage for additive trust behavior (parent Requests, preloaded contexts, CLI fail-fast, apm run propagation).
tests/integration/test_tls_child_runtime.py	Adds integration coverage for managed/foreign Python bootstraps, descendants, Node child propagation, and source mutation stability.
tests/integration/test_tls_frozen_hook.py	Extends frozen-hook env coverage to include additive variables and derived ownership markers.
tests/integration/test_tls_r2_verify.py	Extends verification coverage to include additive variables and derived ownership markers.
tests/integration/test_tls_r3_verify.py	Updates docs-scope integration assertions to match the new runtime-coverage wording and additive guidance.
tests/integration/test_wave6_validation_uninstall_coverage.py	Updates validation guidance assertions to include additive trust recommendations.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Josh Bazar (TameTheGame)]

Merged upstream main at 9cb174b2704c0d770110c9af7bf9ae8e24863f31 into this branch to resolve the conflict introduced by the 0.29.1 release. The changelog preserves the new release entries and keeps additive CA support under Unreleased.

The merge also exposed a new architecture check: two TLS integration tests selected the CLI directly. They now consume the canonical apm_binary_path fixture, so the configured test artifact remains authoritative. The TLS implementation and scope are unchanged.

Validation on Windows:

• Final TLS, child-runtime, script-runner, install TLS guidance, and documentation-scope selection: 223 passed in 42.12 seconds, no skips, using the current branch and an explicitly UAC-elevated Windows process. The earlier run used a filtered, medium-integrity administrator token and failed two symlink fixtures with WinError 1314; both pass with the required process privilege. No Windows policy or application changes were needed.
• After the fixture adaptation, the complete custom-CA integration module passed: 12 tests.
• Ruff lint and formatting, duplication check, auth and architecture boundaries, YAML/file-length/relative-path guards, and diff whitespace check passed.
• Documentation build passed: 124 pages and 1,006 relative links, with no broken links.

These are local results; upstream workflow approval and CI results remain separate. Conflict resolution and the test adaptation were performed with OpenAI Codex assistance.

[Josh Bazar (TameTheGame)]

Resolved the new conflicts after upstream advanced again: retained the corrected 0.29.1 release date and the new MCP staging-path troubleshooting entry alongside this PR's additive TLS guidance. Merged current upstream main; no changes to this PR's TLS implementation or tests were needed.

Validation for this documentation conflict resolution: all 9 TLS documentation-scope tests passed; Ruff lint/format, duplication, architecture and auth boundaries, source guards, and diff whitespace checks passed. Documentation build: 124 pages, 1,006 relative links, no broken links. The earlier 223-test runtime result remains evidence for the previous head; that broader selection was not rerun for this documentation-only resolution.

Performed with OpenAI Codex assistance. Upstream workflow approval and CI results remain separate from these local checks.

[Josh Bazar (TameTheGame)]

Pushed the simplification follow-up in ec035c10.

The change removes redundant inner rollback while retaining recovery around the complete parent and child operations, consolidates the parent fallback, and deletes the duplicate HTTPS test server. It also replaces repeated documentation with a link, removes temporary-file handling from the Bash probe, and corrects the Windows CA-override test expectation. The additive trust and child-runtime scope remain intact. Net change: 93 lines removed across eight files.

Validation for this commit on Windows/Python 3.12.13:

• 364 affected tests passed in 61.04 seconds, with no skips or deselections. All three symlink tests passed with Windows elevation. Two warnings remain in the unchanged lifecycle timeout test's subprocess-reader threads.
• Ruff lint/format, Pylint duplication, architecture/auth checks, CI source guards, and diff whitespace checks passed.
• Documentation built: 124 pages; 1,007 relative links checked, with no broken links.

The fallback regression now forces failure after actual additive-context publication and verifies that real Requests HTTPS still succeeds. Other coverage includes private-CA HTTPS, independent existing-root retention, and Python/Node child propagation.

The PR description has the full validation scope. Linux/macOS execution, a fresh packaged executable, a complete private-registry apm install, and live public-Internet HTTPS were not exercised in this pass. The six upstream workflows currently report action_required, pending repository approval to run.

Review, changes, and validation were performed with OpenAI Codex assistance.

[Daniel Meppiel (danielmeppiel)]

APM Review Panel: ship_with_followups

Additive corporate CA support looks well-scoped and enterprise-ready, with one acceptance-evidence gap before calling #2034 fully demonstrated.

cc Josh Bazar (@TameTheGame) Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

The nine-persona signal is strongly positive on the implementation shape: TLS policy stays centralized in tls_trust, parent trust is additive rather than replacement, Python and Node child propagation use the shared build_child_tls_env seam, and replacement/disable precedence is explicit. Optional APM_EXTRA_CA_DIR is not necessary for this issue. The known Git, Rust/Codex, parent Requests-only fallback, and generic certifi-plus-extra child boundaries are disclosed and match the contributor's documented issue-plan amendment, so they are honest scope limits rather than product faults.

Test evidence should be weighted carefully. The test-coverage pass first hit a local harness mistake by invoking a preinstalled APM, then corrected APM_BINARY_PATH/PYTHONPATH to the PR snapshot and reported 43 targeted TLS/script/lifecycle/runtime tests passing on ec035c1. The synthetic independent-root loopback proof is a sound substitute for live Internet probing because it proves additive retention without third-party dependencies. However, original issue #2034 explicitly named apm install plus apm run; the full private-CA apm install plus default-root fixture remains missing, so #2034 should not be treated as fully demonstrated until that fixture lands or the maintainer explicitly records the install proof as deferred.

CI status is not quality evidence yet: six GitHub workflows are action_required pending repository authorization and only CLA is success, so do not call this green. The practical recommendation is to ship the feature with focused follow-ups tracked: add the install regression trap, clarify the inline-shell trust boundary, and make managed bootstrap refresh failure visible at launch time.

Dissent. I downweight auth's highest-severity inline-shell finding to a docs and regression-trap follow-up. The scenario is real because script_runner.py computes child TLS env before shell inline assignments run, but the evidence is env derivation rather than a demonstrated live handshake failure; shell assignment translation is an explicitly declared shell-owned boundary, inline REQUESTS_CA_BUNDLE itself is honored by Python Requests, and Node does not natively consume APM_DISABLE_TRUSTSTORE.

Aligned with: Secure by default, Multi-harness / multi-host, Pragmatic as npm, Governed by policy

Growth signal. This is a clean enterprise adoption story: trust your corporate CA without breaking public HTTPS. Release framing can confidently market the additive Python/Node path while staying credible by naming Git, Rust/Codex, fallback, and full-install-fixture limits.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	0	1	TLS trust is centralized in tls_trust with scoped child bootstrap and rollback; no blocking architecture issues found.
CLI Logging Expert	0	1	1	Two UX gaps remain: managed llm bootstrap refresh can fail silently, and the startup TLS hard-fail lacks a recovery hint.
DevX UX Expert	0	0	0	No DevX concerns: additive CA, precedence, child coverage, shell semantics, and unset behavior are documented and exercised.
Supply Chain Security	0	0	0	No supply-chain security regressions found in the additive CA implementation.
OSS Growth Hacker	0	0	0	Enterprise onboarding story is clear and scoped: one additive CA knob, runnable docs, and honest runtime boundaries.
Auth Expert	1	0	0	One shell-spawn path lets inline TLS opt-outs inherit APM-derived CA variables; token/auth policy remains isolated.
Doc Writer	0	1	0	Runtime and fallback disclosures match the amended issue; extend the inline-shell caveat to opt-out and replacement controls.
Test Coverage	0	1	0	TLS run/child/default-root tests pass; full private-CA apm install remains unguarded.
Performance Expert	0	0	0	No material performance defect found. Opt-in additive TLS adds bounded local I/O only; warm build_child_tls_env averaged 6.1 ms in a narrow local repro, with no new network RTTs or algorithmic growth.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 3 follow-ups

1. [Test Coverage] Add the full private-CA apm install plus default-root regression fixture. -- Issue Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034 acceptance explicitly names apm install as well as apm run; current targeted tests pass, but the install path is still missing automated proof that a private-CA package source works without replacing normal roots.
2. [Doc Writer + Auth Expert] Broaden the inline-shell TLS boundary documentation and add a focused nested-env regression trap. -- APM resolves child TLS env before shell execution, so inline opt-out or curl replacement assignments cannot remove already-derived Requests/Node settings. Users should be told to set those controls in the environment launching APM rather than inside an apm.yml shell command; do not add fragile cross-platform shell parsing.
3. [CLI Logging Expert] Show a default-level notice when managed llm TLS bootstrap refresh fails at child launch. -- The fallback still carries certifi plus the extra CA and does not disable verification, but a failed managed bootstrap refresh is security-relevant degradation that should not be debug-only.

Architecture

classDiagram
    direction LR
    class CLI {
      <<Entrypoint>>
      +configure_process_tls_trust()
      +cli(ctx, verbose)
    }
    class TLS_TRUST {
      <<Facade>>
      +configure_tls_trust(env) bool
      +build_child_tls_env(base_env, runtime_name) dict
      +explicit_ca_bundle_path(env) str
      +ensure_child_tls_bootstrap(venv_path) bool
    }
    class TLSConfigurationError {
      <<Exception>>
    }
    class ChildCASnapshotStore {
      <<Factory>>
      +_ensure_child_ca_snapshots(bundle_pem) tuple
    }
    class TLSPublicationState {
      <<Memento>>
      +_capture_tls_publication_state()
      +_restore_tls_publication_state(state)
    }
    class ChildBootstrap {
      <<BootstrapAdapter>>
      +_bootstrap()
    }
    class ScriptRunner {
      <<Spawner>>
      +_execute_script_command(command, params) bool
      +_execute_runtime_command(command, content, env) CompletedProcess
    }
    class RuntimeBase {
      <<Streamer>>
      +_stream_subprocess_output(cmd, timeout, env) tuple
    }
    class LLMRuntime {
      <<RuntimeAdapter>>
      +execute_prompt(prompt_content) str
      +is_available() bool
    }
    class ScriptExecutors {
      <<HardenedSessionFactory>>
      +_build_guarded_session()
      +_build_capturing_session()
    }
    class RequestsSession {
      <<ExternalAdapter>>
    }
    class NodeRuntime {
      <<ExternalRuntime>>
    }
    class PythonRequestsChild {
      <<ExternalRuntime>>
    }
    CLI ..> TLS_TRUST : imports before command modules
    TLS_TRUST ..> TLSConfigurationError : raises invalid additive config
    TLS_TRUST *-- ChildCASnapshotStore : freezes extra and certifi-plus-extra
    TLS_TRUST *-- TLSPublicationState : transactional publish/rollback
    TLS_TRUST ..> ChildBootstrap : ships and refreshes
    ScriptRunner ..> TLS_TRUST : build_child_tls_env()
    RuntimeBase ..> TLS_TRUST : default child env
    LLMRuntime ..> TLS_TRUST : runtime_name="llm"
    ScriptExecutors ..> TLS_TRUST : explicit_ca_bundle_path()
    ChildBootstrap ..> TLSPublicationState : local rollback copy
    ChildBootstrap ..> PythonRequestsChild : preserves derived REQUESTS_CA_BUNDLE
    TLS_TRUST ..> NodeRuntime : NODE_EXTRA_CA_CERTS snapshot
    ScriptExecutors ..> RequestsSession : trust_env=False plus explicit verify
    note for TLS_TRUST "Single authority:\nprecedence, validation,\nsnapshots, parent fallback,\nchild env mapping"
    note for TLSPublicationState "Memento-style rollback:\nssl, urllib3, Requests preloaded context"
    class CLI:::touched
    class TLS_TRUST:::touched
    class TLSConfigurationError:::touched
    class ChildCASnapshotStore:::touched
    class TLSPublicationState:::touched
    class ChildBootstrap:::touched
    class ScriptRunner:::touched
    class LLMRuntime:::touched
    class ScriptExecutors:::touched
    classDef touched fill:#fff3b0,stroke:#d47600

Loading

flowchart TD
    A["[I/O] src/apm_cli/cli.py import calls configure_process_tls_trust()"] --> B["src/apm_cli/core/tls_trust.py::configure_tls_trust(env)"]
    B --> C{"has_explicit_ca_override(env)?"}
    C -->|yes| D["Return False; REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE remains replacement authority"]
    C -->|no| E{"APM_DISABLE_TRUSTSTORE truthy?"}
    E -->|yes| F["Return False; OS/additive propagation suppressed"]
    E -->|no| G{"APM_EXTRA_CA_BUNDLE set?"}
    G -->|yes| H["[I/O] _read_extra_ca_bundle(): resolve, fstat, size/ascii/private-key/parser validation"]
    G -->|no| I["[I/O] import truststore; truststore.inject_into_ssl()"]
    H --> I
    I --> J{"truststore injection and _install_additive_ca_context() succeed?"}
    J -->|yes| K["[I/O] publish ssl.SSLContext, urllib3.util.ssl_.SSLContext, requests.adapters._preloaded_ssl_context"]
    J -->|no| L["_restore_tls_publication_state(): rollback loaded ssl/urllib3/Requests globals"]
    L --> M{"extra CA was selected?"}
    M -->|yes| N["[FS] _ensure_child_ca_snapshots(): write certifi-plus-extra fallback under ~/.apm/tls/apm_tls_*/"]
    N --> O["[I/O] set REQUESTS_CA_BUNDLE and APM_REQUESTS_CA_BUNDLE_IS_DERIVED_ADDITIVE in os.environ"]
    M -->|no| P["Return False; bundled certifi fallback only"]
    K --> Q["apm command callbacks can perform HTTPS with OS-plus-extra or OS trust"]
    O --> Q
    Q --> R["src/apm_cli/core/script_runner.py::_execute_script_command() or _execute_runtime_command()"]
    R --> S["src/apm_cli/core/tls_trust.py::build_child_tls_env(env, runtime_name)"]
    S --> T["Clear APM-derived REQUESTS_CA_BUNDLE and NODE_EXTRA_CA_CERTS markers before recomputing"]
    T --> U{"disable or genuine Requests/curl replacement present?"}
    U -->|yes| V["[EXEC] subprocess.run(..., env=child) preserves operator-owned replacement and suppresses derived Node mapping"]
    U -->|no| W["[FS] validate APM_EXTRA_CA_BUNDLE again; write/reuse extra-only and certifi-plus-extra snapshots"]
    W --> X["[EXEC] Python child receives REQUESTS_CA_BUNDLE=certifi-plus-extra snapshot"]
    W --> Y["[EXEC] Node child receives NODE_EXTRA_CA_CERTS=extra-only snapshot unless native value is non-empty"]
    W --> Z{"runtime_name == 'llm'?"}
    Z -->|yes| AA["[FS] _refresh_managed_llm_tls_bootstrap(): ensure_child_tls_bootstrap(~/.apm/runtimes/llm-venv)"]
    Z -->|no| AB["No managed bootstrap refresh"]

Loading

sequenceDiagram
    participant User
    participant CLI as src/apm_cli/cli.py
    participant TLS as src/apm_cli/core/tls_trust.py
    participant Runner as src/apm_cli/core/script_runner.py
    participant Child as Python/Node child process
    User->>CLI: APM_EXTRA_CA_BUNDLE=/corp.pem apm run tls-probe
    CLI->>TLS: configure_process_tls_trust()
    TLS->>TLS: _read_extra_ca_bundle() and configure_tls_trust()
    alt truststore publication succeeds
        TLS-->>CLI: OS trust plus additive CA published
    else publication fails after validation
        TLS->>TLS: _restore_tls_publication_state()
        TLS->>TLS: _ensure_child_ca_snapshots()
        TLS-->>CLI: Requests fallback env uses certifi-plus-extra snapshot
    end
    CLI->>Runner: run command callback
    Runner->>TLS: build_child_tls_env(env, runtime_name)
    TLS->>TLS: clear derived markers, revalidate source, create stable snapshots
    TLS-->>Runner: child env with REQUESTS_CA_BUNDLE and/or NODE_EXTRA_CA_CERTS
    Runner->>Child: subprocess.run(..., env=child_env)
    Child-->>Runner: HTTPS uses frozen additive trust bytes

Loading

Recommendation

Ship the additive CA feature with the three follow-ups above tracked. Treat the code and docs as aligned with the amended issue plan, but do not close the loop on #2034 as fully demonstrated until the private-CA apm install fixture is added or explicitly deferred by the maintainer; describe current workflow status as action_required, not green.

---

Full per-persona findings

Python Architect

• [nit] Architecture pattern note: current TLS owner shape is sufficient. at src/apm_cli/core/tls_trust.py:452
  Design patterns; Used in this PR: Facade / single-authority module -- src/apm_cli/core/tls_trust.py owns trust precedence, additive validation, child snapshots, parent fallback, and child env mapping through configure_tls_trust() and build_child_tls_env().; Used in this PR: Memento-style transactional rollback -- _capture_tls_publication_state() and _restore_tls_publication_state() keep process-wide ssl, urllib3, and Requests publication reversible when additive context installation fails.; Used in this PR: Factory -- _ensure_child_ca_snapshots() creates content-addressed extra-only and certifi-plus-extra artifacts for children rather than letting each spawn path write its own TLS files.; Pragmatic suggestion: none -- splitting this into a registry or strategy hierarchy would add indirection without a third independent TLS policy consumer; keep extending tls_trust as the canonical owner.
  Suggested: Keep future TLS precedence, snapshot, and rollback changes routed through tls_trust; the existing architecture boundary guard already confines truststore.inject_into_ssl() to tls_trust and the child bootstrap.

CLI Logging Expert

• [recommended] Warn when a managed llm launch cannot refresh its TLS bootstrap. at src/apm_cli/core/tls_trust.py:679
  src/apm_cli/core/tls_trust.py degrades launch-time refresh failure to a debug-only line even though the PR and docs promise that APM refreshes the managed llm bootstrap before managed launches. In a reproduced failure where ensure_child_tls_bootstrap returned False, build_child_tls_env({}, runtime_name='llm') emitted no default-level warning and proceeded. That turns a trust-store regression into a later child TLS failure with no immediate guidance, which is the wrong default for a security-relevant runtime degradation.
  Suggested: Mirror runtime_manager._install_llm_tls_bootstrap's yellow warning here, or route both paths through one shared formatter, so a failed launch-time refresh tells the user to re-run apm runtime setup llm, use Python 3.10+, or set PIP_CERT before the child command continues.
  Proof (manual only): (no test ref) -- proves: A managed llm launch can lose the advertised bootstrap refresh with no default-level operator guidance. [secure-by-default,devx]
• [nit] Add a one-line recovery hint to the early Click TLS configuration error. at src/apm_cli/cli.py:163
  src/apm_cli/cli.py raises ClickException(str(_TLS_BOOTSTRAP_ERROR)) before any command callback runs. For a mis-set APM_EXTRA_CA_BUNDLE, the surfaced text is only the raw diagnosis, for example 'APM_EXTRA_CA_BUNDLE path does not exist: ...'. That explains what is wrong but not what to do next, which is a poor first-run experience for a startup-blocking error.
  Suggested: Append one short fix hint such as 'Unset APM_EXTRA_CA_BUNDLE or point it at a readable certificate-only PEM; see SSL / TLS issues.'

DevX UX Expert

No findings.

Supply Chain Security

No findings.

OSS Growth Hacker

No findings.

Auth Expert

• [blocking] Inline shell TLS opt-outs do not clear APM-derived child trust variables. at src/apm_cli/core/script_runner.py:196
  When APM_EXTRA_CA_BUNDLE is set in the environment that launches APM, the shell=True apm run path builds REQUESTS_CA_BUNDLE and NODE_EXTRA_CA_CERTS before the shell applies inline assignments such as APM_DISABLE_TRUSTSTORE=1 or REQUESTS_CA_BUNDLE=/replacement.pem. The direct child can therefore still inherit APM-derived CA trust even though the command explicitly opted out or selected replacement trust, violating the documented replacement/disable precedence and the exact-path ownership invariant for derived values.
  Suggested: Add a regression test with parent APM_EXTRA_CA_BUNDLE plus an inline shell APM_DISABLE_TRUSTSTORE/REQUESTS_CA_BUNDLE assignment, then either clear derived CA variables for that shell command before exec or document and enforce that shell-inline trust policy is unsupported by failing closed instead of silently keeping the derived bundle.
  Proof (manual only): (no test ref) -- proves: A direct apm run shell child can receive additive CA trust after the command-level disable is applied. [secure-by-default,governed-by-policy,multi-harness-support]

Doc Writer

• [recommended] Explain that inline opt-out and curl replacement cannot undo precomputed child trust at docs/src/content/docs/troubleshooting/ssl-issues.md:106
  The caveat explains only that an inline additive assignment is not translated for Node. With APM_EXTRA_CA_BUNDLE exported before apm run, script_runner.py:192-196 derives the environment before the shell evaluates assignments; tls_trust.py:729-739 has already populated REQUESTS_CA_BUNDLE and NODE_EXTRA_CA_CERTS. Consequently, an apm.yml shell command such as 'APM_DISABLE_TRUSTSTORE=1 node probe.js' still inherits the derived Node CA, and 'CURL_CA_BUNDLE=/replacement.pem python probe.py' leaves an ordinary Requests child using the higher-priority derived REQUESTS_CA_BUNDLE. The documented precedence can therefore mislead users attempting per-script opt-out or replacement. A search of the docs and packaged guidance found no explanation of these cases. This is missing operational guidance for the deliberately retained shell boundary, not a request to parse shell commands or a demonstrated verification bypass.
  Suggested: Replace the existing inline-assignment sentence with a concise explanation that APM resolves precedence before shell execution. Tell users to set opt-out/replacement controls in the environment launching APM, for example 'APM_DISABLE_TRUSTSTORE=1 apm run probe' on POSIX, rather than inside the apm.yml shell command. Explicitly note that inline assignments do not remove already-derived Requests/Node settings.

Test Coverage

• [recommended] Add a full-install private-CA regression trap. at src/apm_cli/cli.py:24
  Issue Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034 acceptance names both apm install and apm run, but the Scenario Evidence table maps only parent Requests plus apm run/child-runtime probes. I searched tests/integration and tests for install plus APM_EXTRA_CA_BUNDLE/private_ca_https_server/private CA overlap; the only install hits are docs/guidance or unrelated local install scenarios, and there is no apm_binary_path/CliRunner install test that drives APM_EXTRA_CA_BUNDLE through a private-CA HTTPS install fixture while also proving the default roots remain usable. The affected suite I ran passed once APM_BINARY_PATH was pinned to the snapshot-importing venv script, so this is a missing evidence gap, not a demonstrated branch regression.
  Suggested: Add tests/integration/test_tls_install_custom_ca.py::test_apm_install_trusts_private_ca_and_retains_default_root using synthetic loopback roots, APM_BINARY_PATH=/.venv/bin/apm, PYTHONPATH=src, and no live Internet.
  Proof (test MISSING at): tests/integration/test_tls_install_custom_ca.py::test_apm_install_trusts_private_ca_and_retains_default_root -- proves: apm install can consume a private-CA HTTPS package source via APM_EXTRA_CA_BUNDLE without replacing the normal/default trust roots. [secure-by-default,devx]
  assert install.returncode == 0 and private_package_installed and default_root_probe.returncode == 0

Performance Expert

No findings.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}

[Daniel Meppiel (danielmeppiel)]

Tahnk you for this! Please fix CodeQL vulnerabilities and the panel identified blockers + recommendations/followups

[Josh Bazar (TameTheGame)]

Thanks, Daniel Meppiel (@danielmeppiel). Pushed e9e0301a to address the CodeQL findings and the panel's follow-ups:

• CodeQL: both flagged test client contexts now explicitly require TLS 1.2 or newer. Certificate verification, hostname checking, and the wrong-server-identity rejection test remain enabled.
• Install acceptance: added test_apm_install_trusts_private_ca_and_retains_default_root. The real source CLI installs an actual package from a private-CA HTTPS registry and separately proves an independent default root remains usable. Five fresh-project cases include rejection without the extra CA and rejection when replacement-only trust excludes the default root. Successful cases verify the metadata/download requests, package bytes, lockfile, and deployed instructions. This uses synthetic loopback roots and test-process certifi seeding, with no machine trust changes or transport/resolver mocks.
• Shell boundary: documented that additive, disable, and replacement controls must be set before APM starts deriving child settings. Six real-shell cases cover direct versus nested APM execution, inline disable/Requests/curl controls, and preservation of operator-owned Node settings. Shell parsing remains unchanged, as the panel recommended.
• Launch guidance: failed managed llm bootstrap refresh now gives an actionable default-level warning while preserving the verified additive fallback. Success stays quiet. Invalid startup bundles also include a short recovery hint and troubleshooting link.

Validation on Windows/Python 3.12.13: 373 passed, 2 warnings in 83.30 seconds, with no skips or deselections. All three symlink tests passed with elevation; the two warnings are from the unchanged lifecycle timeout test's reader threads. Required Ruff/format, duplication, architecture/auth, CI source guards, and whitespace checks passed. Documentation built 124 pages and checked 1,031 relative links without errors.

The branch includes upstream main at 1cab81dc and is conflict-free. All six workflow runs on e9e0301 currently report action_required, so upstream validation and CodeQL clearance are still pending.

Would you mind approving the new workflows, reapply panel-review, and re-review this revision? GitHub isn't allowing me to request reviewers or add the label. The PR description and #2034 have the updated acceptance evidence.

[Copilot]

🟡 Changes recommended

The new CHANGELOG entry does not end with a required PR reference (#PR_NUMBER) per repo changelog rules.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 29/29 changed files
• Comments generated: 1
• Review effort level: Lite

[Sergio Sisternes (sergio-sisternes-epam)]

Thank you for contributing this pull request and for the follow-up
on #2034. APM starts with an issue, not an implementation
(https://github.com/microsoft/apm/blob/main/CONTRIBUTING.md).

#2034 already describes the user problem, but it currently lacks
status/accepted after the 2026-09-12 acceptance withdrawal. Please
wait for a maintainer to review and accept the bounded scope on that
issue before further implementation. This PR is labelled
status/deferred until that happens.

This recommendation is not merge approval and does not request a
review-panel or merge-worker run.

---

Generated by autopilot-pr-triage-worker. This comment is AI-generated and may contain errors.

[Josh Bazar (TameTheGame)]

Thanks Daniel Meppiel (@danielmeppiel). Pushed 73ae7920 to align this PR with the October 2 approval.

• Scope: APM_EXTRA_CA_BUNDLE now covers APM's own package-management HTTPS. Removed the new Python/Node execution-child propagation, CA snapshot files, ownership markers, nested-shell handling, and managed-runtime refresh. The child bootstrap, script runner, and llm runtime match current upstream.
• Trust behavior: default roots remain available; explicit Requests/curl replacement and truststore opt-out controls retain precedence. Both Requests and stdlib metadata HTTPS retain the extra CA on fallback. Certificate and hostname verification remain enabled, invalid selected input fails clearly, and failed publication restores the previous TLS state.
• Acceptance: ten fresh-project cases exercise the real source apm install with normal OS trust and forced truststore unavailability. They cover private-CA success, independent default-root retention, and untrusted/replacement-only rejection, checking requests, package bytes, lockfile, and deployed instructions. Real child probes confirm no new additive trust is exported. These use synthetic loopback roots and test-process trust seeding, with no machine trust edits or mocked transports/resolver.
• Validation: 363 affected tests passed on Windows/Python 3.12.13, with no skips or deselections; all three symlink tests ran with elevation. The two warnings are from the unchanged lifecycle timeout test. Final focused checks passed 28 tests, and repository quality contracts passed 64. Required lint/format, duplication, architecture/auth, source guards, assertion/duplicate ratchets, and whitespace checks passed. Docs built 125 pages, checked 1,071 links, and matched all 34 public commands to reference pages.

The PR is conflict-free against upstream main at 18c4c43c. Its description now uses the current approval record and the narrowed acceptance evidence. The contribution is down from 29 changed files to 21, with 953 fewer net added lines.

The six new CI/docs/CodeQL/spec/merge/NOTICE workflows require maintainer approval. Could you approve those runs and re-review this revision? The status/deferred label also still reflects the earlier hold; the linked October 2 approval is the current scope record.