Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README-ADMIN.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ report is scoped to just those workspaces.
- **`--no-open`**: Don't automatically open the generated HTML file after generation.
- **`--csv-dir DIR`**: Also write Glue-ready CSV fact tables (`growth_users.csv`, `growth_workspaces.csv`, `growth_org_kpis.csv`) into `DIR`. `DIR` is created if it doesn't exist.
- **`--no-html`**: Skip the HTML report. Requires `--csv-dir` (otherwise there is nothing to write, and the command errors out).
- **`--mpm`**: Include MPM presence — whether each workspace uses MPM and how many models it monitors (registry models flagged `is_monitored`). Chargeback has no MPM data, so this is collected from `/api/mpm/v3/workspaces` (one call, covering the workspaces the API key's user belongs to) and, for the rest, the REST v2 model registry (one request per registry model). The key's user must be an **organization admin** for this to be complete. That's a different check from the one chargeback applies: the chargeback endpoint accepts server admins, while the registry lets organization admins read every workspace. For a workspace the user isn't a member of, a key that isn't an organization admin gets only the workspace's *public* models, so private monitored models are silently missed and the workspace still counts as checked. Off by default because of the extra requests.
- **`--mpm`**: Include MPM presence — whether each workspace uses MPM and how many models it monitors (registry models flagged `is_monitored`). Chargeback has no MPM data, so this is collected from `/api/mpm/v3/workspaces` (one call, covering the workspaces the API key's user belongs to) and, for the rest, the REST v2 model registry (one request per registry model). The key's user must be an **organization admin** (or a member of every workspace) for this to be complete. That's a different check from the one chargeback applies: the chargeback endpoint accepts server admins, while the registry lets organization admins read every workspace. For a workspace the user isn't a member of, a key that isn't an organization admin gets only the workspace's *public* models, so private monitored models are silently missed and the workspace still counts as checked. Off by default because of the extra requests.
- **`--chargeback-report FILE`**: Read the chargeback report from a local JSON file (as saved by `cometx admin chargeback-report`) instead of calling the chargeback endpoint. Useful for CSV-only pipelines that already have a saved snapshot, or for re-running without re-fetching it. Note this skips the *chargeback* request only — the report still queries `/api/admin/service-accounts` to classify accounts, and falls back to a name-pattern heuristic if that request fails (`service_account_source` records which was used).

### The two time concepts
Expand Down Expand Up @@ -223,7 +223,7 @@ The report generates a single self-contained HTML file containing:

- **Chargeback is required.** The whole report is derived from the admin chargeback report; without admin access the command errors out (non-zero exit).
- **Workspace "created" is a proxy** — the earliest member `createdAt` in that workspace, since chargeback has no workspace-creation timestamp. The added-vs-deleted "deleted" series is also a best-effort proxy (all members removed) and typically reads ~0.
- **"Total projects" counts EM projects only** — chargeback's per-workspace `projects[]` covers Experiment Management. Opik projects and MPM aren't represented there (Opik appears only as a per-user span count; MPM is absent), so the platform mix uses an Opik per-user proxy and excludes MPM.
- **"Total projects" counts EM projects only** — chargeback's per-workspace `projects[]` covers Experiment Management. Opik projects and MPM aren't represented there (Opik appears only as a per-user span count; MPM comes only from `--mpm`, below), so the platform mix uses an Opik per-user proxy and excludes MPM.
- **MPM comes from `--mpm`, not chargeback.** It reflects which models are monitored *now*, so past months can't be reconstructed; a trend builds up from monthly `--csv-dir` exports. A workspace whose MPM lookup fails is reported as unknown (empty in the CSV), never as zero, and the report says how many workspaces were checked. Counts are only complete when the key's user is an organization admin or a member of every workspace (see `--mpm`). `nb_models_registered` from the monthly usage report is not used: it counts every registry model and overstates MPM adoption.
- **The people layer degrades independently.** If the chargeback payload parses but a section's inputs are missing, a warning is printed and only that section is dropped — the rest of the report still generates.

Expand Down Expand Up @@ -348,7 +348,8 @@ cometx admin growth-report --csv-dir ./out
# CSV-only, no HTML
cometx admin growth-report --csv-dir ./out --no-html

# Regenerate CSVs from a saved chargeback snapshot, no API call
# Regenerate CSVs from a saved chargeback snapshot (skips the chargeback
# request; service-accounts is still queried)
cometx admin growth-report --chargeback-report report.json --csv-dir ./out
```

Expand Down
28 changes: 22 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -607,7 +607,10 @@ cometx admin growth-report [WORKSPACE ...]
* `--exclude-personal` - Drop workspaces whose name matches `--personal-pattern` from the chargeback data (default: off; no effect without `--personal-pattern`).
* `--personal-pattern REGEX` - Regex used with `--exclude-personal` to identify personal-workspace names to drop, e.g. `'^user-'` (default: none).
* `--no-open` - Don't automatically open the generated HTML file after generation.
* `--mpm` - Include MPM presence (monitored models per workspace), collected from the MPM and model-registry APIs. The key's user must be an organization admin (a separate check from chargeback's server-admin one) or private models in workspaces they aren't a member of are silently missed; off by default because it makes one request per registry model in workspaces the key's user isn't a member of.
* `--csv-dir DIR` - Also write Glue-ready CSV fact tables (`growth_users.csv`, `growth_workspaces.csv`, `growth_org_kpis.csv`) into `DIR` (created if missing). See [CSV export](README-ADMIN.md#csv-export) for the schema and the S3/Glue layout.
* `--no-html` - Skip the HTML report. Requires `--csv-dir`; with neither, there is nothing to write and the command errors out.
* `--mpm` - Include MPM presence (monitored models per workspace), collected from the MPM and model-registry APIs. The key's user must be an organization admin (a separate check from chargeback's server-admin one) or a member of every workspace; otherwise private models in workspaces they aren't a member of are silently missed. Off by default because it makes one request per registry model in workspaces the key's user isn't a member of.
Comment thread
dsblank marked this conversation as resolved.
* `--chargeback-report FILE` - Read the chargeback report from a local JSON file (as saved by `cometx admin chargeback-report`) instead of calling the chargeback endpoint. This skips the chargeback request only; `/api/admin/service-accounts` is still queried to classify accounts.

**Two time concepts:**
* `--units` controls chart *granularity* — every chart shows the full all-time history bucketed at this resolution.
Expand All @@ -619,19 +622,20 @@ proxied from each workspace's earliest member `createdAt`.

**Report sections:** an **Organization overview (chargeback)** section (org-wide
KPIs, workspace platform mix, total-vs-active and added-vs-deleted charts, and a
by-workspace table), a **Users** section (Total / Active / Active % / New-in-window
by-workspace table; with `--mpm`, also MPM workspaces / Monitored models KPIs and
an MPM models column), a **Users** section (Total / Active / Active % / New-in-window
Comment thread
dsblank marked this conversation as resolved.
KPIs plus over-time charts), a **Leaderboards** section (top-N and active-aware
bottom-N workspaces by experiments/projects and users by Opik spans / EM activity,
exact from chargeback), and a **Personal vs Service accounts** split (service
bottom-N workspaces by experiments/projects, and by MPM monitored models with
`--mpm`, and users by Opik spans / EM activity), and a **Personal vs Service accounts** split (service
accounts from the admin API, with a labeled regex fallback). Each people section
degrades independently: if its inputs are missing, a warning is printed and only
that section is dropped.

**Caveats:**
* **Chargeback is required** — the whole report is derived from the admin chargeback report; without admin access the command errors out.
* **Workspace "created" is a proxy** — the earliest member `createdAt`, since chargeback has no workspace-creation timestamp. The added-vs-deleted "deleted" series is also a best-effort proxy (all members removed) and typically reads ~0.
* **"Total projects" counts EM projects only** — chargeback's per-workspace `projects[]` covers Experiment Management; Opik projects and MPM aren't represented (Opik appears only as a per-user span count; MPM is absent).
* **MPM needs `--mpm`.** Chargeback has no MPM data; `--mpm` collects monitored models per workspace from the MPM and model-registry APIs (see README-ADMIN.md).
* **"Total projects" counts EM projects only** — chargeback's per-workspace `projects[]` covers Experiment Management; Opik projects and MPM aren't represented there (Opik appears only as a per-user span count; MPM comes only from `--mpm`, below).
* **MPM needs `--mpm`.** Chargeback has no MPM data; `--mpm` collects monitored models per workspace from the MPM and model-registry APIs. It reflects which models are monitored *now*, so a trend builds up only from regular `--csv-dir` exports. A workspace whose lookup fails is reported as unknown, never as zero (see [README-ADMIN.md](README-ADMIN.md#growth-report)).

**Examples:**
```
Expand All @@ -648,6 +652,18 @@ cometx admin growth-report --no-open --output growth.html
# workspaces named like "user-..."
cometx admin growth-report --active-window 30d --leaderboard-top-n 10 \
--exclude-personal --personal-pattern '^user-'

# Write Glue-ready CSVs alongside the HTML report
cometx admin growth-report --csv-dir ./out

# CSV-only, no HTML
cometx admin growth-report --csv-dir ./out --no-html

# Regenerate CSVs from a saved chargeback snapshot
cometx admin growth-report --chargeback-report report.json --csv-dir ./out

# Include MPM presence (monitored models per workspace)
cometx admin growth-report --mpm --csv-dir ./out
```

#### gpu-report
Expand Down
9 changes: 7 additions & 2 deletions cometx/cli/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -479,13 +479,18 @@ def get_parser_arguments(parser):
One or more workspaces to run the growth report for.
If not provided, all workspaces are included.

Requires an ADMIN API key: the report is built entirely from the admin
chargeback report. Without admin access the command exits with an error.
Requires an admin user's API key: the report is built entirely from the admin
chargeback report, which accepts only server admins and, on self-hosted
installs, organization admins. Workspace roles such as Manage don't count.
Without admin access the command exits with an error.

Output:
Generates a self-contained HTML page with the organization overview, users,
personal-vs-service, and leaderboard sections, all derived from the admin
chargeback report and broken down by workspace/department.
With --csv-dir, also writes Glue-ready CSV fact tables (growth_users.csv,
growth_workspaces.csv, growth_org_kpis.csv); add --no-html for CSV-only
output. With --mpm, adds MPM presence (monitored models per workspace).

Examples:
cometx admin growth-report
Expand Down
Loading