Skip to content

Capture Copilot assistant usage events#21

Merged
maryliag merged 1 commit into
open-telemetry:mainfrom
trask:copilot-usage-assistant-events
Jun 26, 2026
Merged

Capture Copilot assistant usage events#21
maryliag merged 1 commit into
open-telemetry:mainfrom
trask:copilot-usage-assistant-events

Conversation

@trask

@trask trask commented Jun 26, 2026

Copy link
Copy Markdown
Member

Last PR didn't report everything.

Capture token usage from Copilot CLI assistant.usage JSONL events, including cached and reasoning token fields, so the pull request dashboard reports exact token totals when the CLI emits its current usage event shape.

@trask trask requested a review from Copilot June 26, 2026 18:59
@trask trask marked this pull request as ready for review June 26, 2026 19:00
@trask trask requested a review from a team as a code owner June 26, 2026 19:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the pull-request-dashboard telemetry scripts to capture additional Copilot token-usage metrics (cache read, cache write, and reasoning tokens) and to recognize the Copilot CLI's assistant.usage JSONL event as a usage source. Previously these tokens and the dedicated usage event were not captured, so the dashboard under-reported token totals. The change is well-contained: it adds the three new fields wherever existing token fields are handled, keeping the ingest (classification.py) and aggregation/reporting (dashboard.py) layers in sync.

Changes:

  • Added cache_read_tokens, cache_write_tokens, and reasoning_tokens to the usage field aliases and normalization in classification.py, and mirrored them in the dashboard's empty/aggregate/print helpers in dashboard.py.
  • Reworked parse_copilot_jsonl to read evt["data"] once and treat an assistant.usage event's data as the usage payload, with a fallback to data["usage"].

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.github/scripts/pull-request-dashboard/classification.py Adds new token-field aliases, normalizes them, and parses assistant.usage events as a usage source.
.github/scripts/pull-request-dashboard/dashboard.py Mirrors the three new token fields in the usage initializer, aggregator, and summary printer.

I did not identify concrete, objective defects in the diff: the new fields are consistently threaded through both files, missing keys are guarded with or 0, and per-event usage selection avoids within-event double counting. The one aspect I could not verify is the external contract of the Copilot CLI's assistant.usage event (e.g., whether it is emitted as a cumulative running total or alongside per-message usage), which the cross-event summation in parse_copilot_jsonl depends on for correct totals.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@maryliag maryliag merged commit 2556738 into open-telemetry:main Jun 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants