[OPIK-8411] MPM presence in growth-report via --mpm (OPIK-8411)

[dsblank]

Adds MPM adoption to cometx admin growth-report for OPIK-8411: which workspaces use MPM and how many models each one monitors, next to the existing EM and Opik stats. It also fixes the admin API URL and the chargeback error messages, which were blocking growth-report on servers such as mpm-demo.

cometx admin growth-report --mpm --csv-dir ./out

How --mpm collects the data

The chargeback report has no MPM data, and the backend is out of scope, so --mpm gathers it from existing endpoints. It merges mpmEnabled / monitoredModels into each chargeback workspace before the report is built:

1. /api/mpm/v3/workspaces: one call covering every workspace the API key's user belongs to. The path is deliberately fixed: it's a backend-react route, so it resolves the same way on Cloud and on chart deployments.
2. REST v2 model registry for the other workspaces: registry-model?workspaceName= lists the models, then registry-model/details gives each model's monitored flag. This is one request per model, so the calls run in parallel (8 workers).

Both sources apply the same filter server-side: is_monitored, not pipeline-generated, no ref_production_model_id, not deleted (RegistryQueries.getMonitoredRegistryModelsForWorkspacesSql). When workspaces are named on the command line, only those are checked. --mpm is off by default because of the per-model requests.

Nothing unknown is reported as zero or as complete:

• A workspace whose lookups fail, or whose flag can't be read, is unknown, not zero. A failed lookup also removes any MPM fields already in the input, so stale values aren't passed through.
• Partial totals are labeled. The MPM-workspaces percentage is over the workspaces actually checked ("100% of 1 checked; 1 unknown"). A model total missing any workspace is shown as a lower bound ("≥ N", "MPM checked in K/M workspaces").
• A failed mpm/v3/workspaces call is classified, not swallowed (review feedback). The results are refused (401/403), not_found (404: MPM not installed or not routed) and error. Each prints a warning, is shown in the HTML notes, and is recorded in the CSV. refused also warns that counts may be low (see Access below).
• nb_models_registered is not used, because it counts every registry model (the ticket's caveat).

Output

• HTML report
  • Organization overview: MPM workspaces and Monitored models cards, and an MPM models column in the by-workspace table.
  • Leaderboards: top/bottom workspaces by MPM monitored models.
  • The platform-mix chart is unchanged. Its hint says whether MPM was collected, and how.
• growth_workspaces.csv: new mpm_enabled (1/0, so SUM() counts MPM workspaces) and num_monitored_models columns, appended at the end so existing columns keep their order. Both are empty when that workspace's MPM status is unknown.
• growth_org_kpis.csv:
  • mpm_workspaces and total_monitored_models: exact only when mpm_workspaces_unchecked is 0.
  • mpm_workspaces_unchecked: workspaces whose model count is unknown. Emitted even when every lookup fails.
  • mpm_member_lookup: a label metric (ok / refused / not_found / error), so a dashboard can tell "MPM absent" from "auth refused".

Access: who counts as an admin

The two parts of this report use different permission checks:

• Chargeback (@AdminApiKey, ApiKeyAuthFilter): the key's user must be in the server's admin user list or, on self-hosted installs, an organization admin. Workspace roles such as Manage don't count.
• --mpm registry route: organization admins can read every workspace. For a workspace the user isn't a member of, a user who isn't an organization admin gets only its public models, with no error. Private monitored models are then missed. This is documented in --help, both READMEs, and the module docstring. mpm_member_lookup=refused flags the most likely case, but the code can't detect it directly.

Also fixed

• Admin URLs under the SDK's /clientlib root. admin_api_url kept the /clientlib segment that the SDK's comet.url_override ends with. So chargeback, service-accounts, migrate-users and the MPM call all went to /clientlib/api/..., which is a 404 on mpm-demo. It now drops a trailing /clientlib segment and keeps any deployment prefix in front of it (/comet/clientlib becomes /comet), matching smoke_test.

• Accurate chargeback errors. Every chargeback failure used to print "requires an admin API key", including that 404. The message now depends on the failure:

  Failure	Message
  401 / 403	requires an admin user's API key: server admin, or organization admin on self-hosted; workspace roles such as Manage don't count
  404	could not find the chargeback endpoint at <url>; check the server URL (not an API-key problem)
  non-JSON response	likely an SSO or proxy login page
  anything else	could not fetch the chargeback report, without blaming the key

  The detail shown is HTTP <status>: <server message>. Before, it was the SDK exception text, which included the first 5 characters of the API key and a truncated URL.

Limitations

• Current state only. The monitored flag says what's monitored now, so past months can't be reconstructed. A trend builds up from monthly --csv-dir exports.
• Prediction volume is out of scope per the ticket (it would need Druid).

Testing

• 340 unit tests pass, and pre-commit hooks are clean on the changed files. tests/unit/test_admin_growth_mpm.py covers:

  • member-workspace coverage and the registry fallback
  • listing and detail failures, and a missing flag
  • both flag spellings
  • scoping to named workspaces
  • stale fields on a failed lookup
  • partial-coverage labels
  • member-lookup results (401, 403, 404, 502, network, unexpected response)
  • --mpm on and off through GrowthReporter.build

  Other test files cover the URL handling and each chargeback error message.

• Live on mpm-demo.dev.comet.com (comet.mpm.enabled), full run with an admin key (growth-report --mpm --csv-dir, no URL override): exit 0 in about 4s.

  • 10 workspaces: 1 answered by mpm/v3/workspaces, 9 via the registry, mpm_member_lookup=ok, mpm_workspaces_unchecked=0.
  • 4 MPM workspaces (40%) with 18 monitored models, shown as complete in the cards and the CSV.
  • The registry returned private models in 3 workspaces the key's user isn't a member of, so nothing was missed.
  • The details response spells the flag monitored, not isMonitored. Both are accepted.
  • Before the admin role was granted: the non-admin key got the 401 admin-key message, where it previously got a 404 from the /clientlib URL.

• Not tested live: the refused and not_found lookup results (unit tests only), and a SageMaker deployment. Review feedback traced the SageMaker auth path through api._client and found no code needed.

🤖 Generated with Claude Code

[baz-reviewer]

Baz Summary

Add optional --mpm collection to cometx admin growth-report, combining MPM workspace responses and registry model details with chargeback data to report monitored-model adoption, partial coverage, CSV metrics, HTML KPIs, and leaderboards. Correct admin URL construction and chargeback diagnostics, while applying formatting cleanup across CLI, framework, example, and reporting files.

graph LR
admin_("admin"):::modified
generate_growth_report_("generate_growth_report"):::modified
GrowthReporter_build_("GrowthReporter.build"):::modified
GrowthReporter_add_mpm_presence_("GrowthReporter._add_mpm_presence"):::added
fetch_mpm_presence_("fetch_mpm_presence"):::added
MPM_API_("MPM_API"):::added
registry_monitored_models_("_registry_monitored_models"):::added
apply_mpm_presence_("apply_mpm_presence"):::added
admin_ -- "Forwards --mpm to enable monitored-model collection in reports." --> generate_growth_report_
generate_growth_report_ -- "Passes MPM configuration so build conditionally enriches chargeback workspaces." --> GrowthReporter_build_
GrowthReporter_build_ -- "Collects selected workspace MPM data before assembling report output." --> GrowthReporter_add_mpm_presence_
GrowthReporter_add_mpm_presence_ -- "Requests monitored model lists and returns coverage or lookup status." --> fetch_mpm_presence_
fetch_mpm_presence_ -- "Sends API-key authorization and receives workspace monitored-model records." --> MPM_API_
fetch_mpm_presence_ -- "Falls back to registry listings and per-model monitored flags." --> registry_monitored_models_
GrowthReporter_add_mpm_presence_ -- "Adds mpmEnabled and monitoredModels fields, preserving unknown workspaces." --> apply_mpm_presence_
classDef added stroke:#15AA7A
classDef removed stroke:#CD5270
classDef modified stroke:#EDAC4C
linkStyle default stroke:#CBD5E1,font-size:13px

Loading

Topics

Topic	Details
MPM growth reporting	Collect MPM presence through fetch_mpm_presence, merge monitored models into chargeback workspace records, preserve unknown and partial results, and expose the data through report KPIs, tables, leaderboards, CSV exports, CLI help, documentation, and regression tests. Modified files (11) README-ADMIN.md README.md cometx/cli/admin.py cometx/cli/admin_growth_csv.py cometx/cli/admin_growth_mpm.py cometx/cli/admin_growth_report.py cometx/cli/admin_growth_users.py tests/unit/test_admin_growth_csv.py tests/unit/test_admin_growth_mpm.py tests/unit/test_admin_growth_report.py tests/unit/test_admin_growth_users.py Latest Contributors(2) User Commit Date nimrod@comet.com fix(admin): split the ... October 01, 2026 doug@comet.com fix(admin): classify a... September 30, 2026
Admin API reliability	Correct admin_api_url handling for SDK /clientlib roots and distinguish authentication failures, missing endpoints, malformed responses, and server errors so chargeback and MPM workflows produce accurate, safe diagnostics across deployment prefixes. Modified files (5) cometx/cli/admin_growth_report.py cometx/utils.py tests/unit/test_admin_growth_report.py tests/unit/test_utils.py tests/unit/test_utils_chargeback.py Latest Contributors(2) User Commit Date nimrod@comet.com fix(admin): split the ... October 01, 2026 doug@comet.com fix(admin): classify a... September 30, 2026
Codebase formatting	Apply formatting and import-order cleanup to administrative reports, copy and download utilities, duplicate renaming, and example workflows without changing their behavior, improving consistency and maintainability across the broader CLI and sample surface. Modified files (12) cometx/cli/admin_gpu_app.py cometx/cli/admin_gpu_report.py cometx/cli/copy.py cometx/cli/rename_duplicates.py cometx/framework/comet/download_manager.py examples/00_seed_demo_project.py examples/01_inspect_views.py examples/02_build_project_dashboard.py examples/03_build_experiment_view.py examples/04_migrate_views.py examples/05_standardize_workspace.py examples/06_harvest_panels.py Latest Contributors(2) User Commit Date nimrod@comet.com fix(admin): take an HT... October 01, 2026 chasefortier Add workspace creation... March 30, 2026

  _{Activate to get a short verdict whether this PR is good to go or not}

  _{Activate Skill Maintainer to keep your skills up to date}

  _{This PR would have been improved with Baz Planner - Try it now}

[Nimrod007]

SageMaker: the auth works and needs nothing here, but a failure would be invisible

Checked whether --mpm works under SageMaker Partner App auth, the way the MPM SDK does. It does, and admin_growth_mpm.py needs no SageMaker-specific code — because it calls through api._client rather than raw requests.

The chain: api._client.get(...) → BaseApiClient.get → LowLevelHTTPClient.get → session.get(...), where the session comes from comet_ml's get_comet_http_session(), which sets session.headers["Authorization"] = api_key and then calls setup_http_session_authentication(session). That checks AWS_PARTNER_APP_AUTH and installs session.auth = PartnerAppAuthProvider().get_auth() — the same provider the MPM SDK wires up in comet_mpm/authhook/aws_sagemaker.py. comet_ml carries its own copy at comet_ml/authhook/aws_sagemaker.py.

The explicit headers={"Authorization": api.api_key} is redundant, since the session already sets it, but harmless: requests runs prepare_auth() after prepare_headers(), so the signer still sees the Comet key and relocates it to X-Amz-Partner-App-Authorization with SigV4 in Authorization.

The hardcoded path is correct, and that is worth recording because it reads as a bug. /api/mpm/v3/workspaces looks like an MPM-service route, but @Path("mpm/v3/workspaces") is registered in ReactWebappServerApplication — it is a backend-react route, not an MPM-webapp one. So it resolves through the gateway on Cloud and through nginx's location /api/ on a chart deployment, and making it configurable would break it. Same shape as RECENT_EVENTS_ENDPOINT in the MPM SDK; the MPM query API (api/mpm/v2/...) is the one that genuinely varies by deployment, and this is not that.

Server side, MpmAuthFilter.extractApiKeyFromHeader reads HttpHeaders.AUTHORIZATION and expects a Comet API key or workspace JWT. Nothing in comet-backend handles X-Amz-Partner-App-Authorization, so restoring that header is the AWS proxy's job — which it must already do, since every comet_ml API call under SageMaker depends on the identical round-trip.

One thing I could not verify: the sagemaker package was not available in my environment, so I did not read PartnerAppAuthProvider.get_auth() itself. The relocation behaviour is inferred from the MPM SDK's comment on the sibling get_signed_request, plus the fact that get_comet_http_session sets the Authorization header before installing the signer, which only works if the signer moves it.

---

The one change I would ask for

_fetch_member_workspaces catches bare Exception and returns None, and the module docstring reads that as "MPM disabled on the deployment, network error, unexpected shape". Under SageMaker it also covers a 401, a 403, and a proxy routing miss — and all of them fall through to the registry path rather than surfacing.

That matters because of the second caveat this file already documents: on a non-member workspace, a caller who is not an org admin sees only public models, with no error. So if the MPM endpoint fails and the key is not an org admin, the two silent paths compose into a confident-looking low number rather than an unknown.

This is the failure mode the module sets out to prevent — "a false zero would read as a workspace that stopped using MPM" — and the fallback reintroduces it one level up. Suggest logging the exception at debug or warning, and distinguishing 401/403 from the rest, so a deployment where this never worked is visibly different from one where MPM is simply not installed.

Worth confirming on a deployment with comet.mpm.enabled true (it defaults false in the chart and is orthogonal to sagemaker.enabled) that the report can tell "MPM absent" from "auth refused". Right now it cannot.

[dsblank]

Thanks for tracing the SageMaker path. Agreed on the change you asked for; it's in 0afbeda.

The mpm/v3/workspaces lookup no longer fails silently. It's classified as one of:

Result	When	What happens
ok	the call worked	as before
refused	401 / 403	warning with the URL and status, plus a note that the registry fallback hides private models in non-member workspaces from users who aren't org admins, so counts may be low
not_found	404	warning that MPM may not be installed or routed on this deployment
error	anything else, including an unexpected response shape	warning with the error

The registry fallback still runs in every failure case, but the result is recorded:

• CSV: a new mpm_member_lookup label KPI in growth_org_kpis.csv (metric_text = ok / refused / not_found / error), so a dashboard can tell "MPM absent" from "auth refused".
• HTML: the MPM notes say so, e.g. "MPM API refused this key; counts may be low". That includes the Monitored models card, even when every workspace was checked.

I also recorded in the module docstring why the /api/mpm/v3/workspaces path is fixed (it's a backend-react route), since it reads like a bug.

Tested:

• Unit tests for 401, 403, 404, 502, a network error, an unexpected response shape, the CSV and HTML output, and no KPI when --mpm isn't given. 340 tests pass.
• Live on mpm-demo (comet.mpm.enabled) with an admin key: mpm_member_lookup=ok, 4 of 10 workspaces with 18 monitored models, 0 unchecked. I couldn't reproduce refused or not_found live there, so those are covered only by the unit tests. A check on a SageMaker deployment would still be worthwhile.

I kept the explicit Authorization header. As you say, it's redundant but harmless, and it matches the other admin calls (fetch_chargeback_report, service-accounts).

Baz dismissed its prior approval because a re-review found new findings.