feat(analytics): org model-usage analytics — ledger fields, telemetry dimensions, API + dashboard - #3491
Open
benjaminshafii wants to merge 12 commits into
Open
feat(analytics): org model-usage analytics — ledger fields, telemetry dimensions, API + dashboard#3491benjaminshafii wants to merge 12 commits into
benjaminshafii wants to merge 12 commits into
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Member
Author
Photo roll — two-members-visibly-drive-default-and-manual-model-analytics-end-to-end — 5/5 frames passed · 2 facts✅ 5/5 frames passed · 2 facts · 12 expectations passed · 0 failed ✅ PASS — 1. The Models section visibly says Sessions by model and No model usage yet
✅ PASS — 2. The composer visibly shows Analytics Default as the selected model
✅ PASS — 3. The open model picker visibly offers Analytics Manual
✅ PASS — 4. The composer visibly shows Analytics Manual as the selected model
ℹ️ FACT — 5. Two real model sessions reached org analytics[{"id":"analytics-witness/model-default","label":"analytics-witness/model-default","sessions":1},{"id":"analytics-witness/model-manual","label":"analytics-witness/model-manual","sessions":1}]
ℹ️ FACT — 6. Default and manual selection totals are distinct{"default":1,"manual":1}
✅ PASS — 7. The Sessions by model chart visibly contains bars for analytics-witness/model-default and analytics-witness/model-manual
Roll created 2026-08-04T21:31:24.053Z · Source: |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![diff-warden[bot]](https://avatars.githubusercontent.com/u/118544361?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warden security clearance: clear. No new security issues found in this diff (b4ca3f73e84b3771a79c3bfd05ce8fde8b841f4b). Automated clearance satisfies the required-review gate only — a human still reviews and merges. Analysis run
…analytics # Conflicts: # ee/packages/den-db/drizzle/meta/0052_snapshot.json # ee/packages/den-db/drizzle/meta/_journal.json
Deploying openwork with
|
| Latest commit: |
be92aaf
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://39cc8a99.openwork.pages.dev |
| Branch Preview URL: | https://feat-3483-model-usage-analyt.openwork.pages.dev |
benjaminshafii
enabled auto-merge (squash)
August 4, 2026 21:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3483
What
Org admins get model-usage analytics: which models their team actually uses, whether people stick with the default model or manually switch, and per-request token counts persisted for future spend views.
How (five stages, one per commit)
inference_usage_ledger_entriesgains nullablemodel_id,provider_id,input_tokens,output_tokens,total_tokens(ee/packages/den-db/src/schema/inference.ts, migrationdrizzle/0052_brown_rictor.sql).ee/apps/inference/src/webhooks.ts).model(provider/model) andmodel_selection(default|manual, derived from whether a session model override is set) session dimensions through the existing telemetry path (apps/app/src/react-app/shell/session-route.tsx).GET /v1/telemetry/analyticsreturns amodelsblock: per-model distinct-session counts (30d) and default-vs-manual selection totals; existing enterprise entitlement gate unchanged (ee/apps/den-api/src/routes/telemetry/index.ts).ee/apps/den-web/.../analytics-screen.tsx).Aggregates only — no message content is captured anywhere.
Tests (commands + results)
pnpm --filter @openwork-ee/den-db db:generate+test— migration generated; 13 pass, 3 skip.ee/apps/inference/test/webhooks.test.ts) — extended to assert persisted model/provider/token fields; pass. Full inference suite: 30 pass / 9 fail — the 9 failures are pre-existing on dev (staleopenrouter/fusionfixtures; identical failure set reproduced on a clean dev checkout, which fails 10 — this branch fixes one of them).evals/specs/org-model-analytics.test.ts(pnpm --dir evals run spec specs/org-model-analytics.test.ts) — cold-boots a local Den via@openwork/testkitserver(): ingests sessions withmodel/model_selectiondimensions via the public ingest API, then asserts the admin analytics response aggregatesmodels.usage30dandselection30dcorrectly and that a second org sees none of it (no cross-org leakage). Passed 1/1. Evidence tape published on this PR.Notes
/(needed forprovider/modelvalues).Update: full app-driven demo + a bug it caught
evals/specs/org-model-analytics-e2e.slow.test.ts(stack lane): two real members in the desktop app — one sends with the org default model, the other manually picks a different model via the picker (deterministic witness provider, per-identity token assertions); org analytics then shows both models (API counts asserted + den-web dashboard visibly rendering both bars and the 1/1 default-vs-manual split). Passed; photo roll with screenshots published below (5/5 frames, 2 facts).fetch, which is cross-origin-blocked in Electron — desktop-emitted dimensions never reached Den. Now proxied through Electron main when cross-origin (apps/app/src/app/lib/den-telemetry.ts).apps/desktop/scripts/electron-dev.mjs: opt-inOPENWORK_ELECTRON_SKIP_NATIVE_REBUILD=1for prebuilt native bindings (default behavior unchanged)..slow.test.tsso the app-less PR lane (no MySQL) excludes them, per the lane convention.