Skip to content

Log raw Copilot OTel output#24

Merged
maryliag merged 1 commit into
open-telemetry:mainfrom
trask:copilot-otel-raw-logs
Jun 26, 2026
Merged

Log raw Copilot OTel output#24
maryliag merged 1 commit into
open-telemetry:mainfrom
trask:copilot-otel-raw-logs

Conversation

@trask

@trask trask commented Jun 26, 2026

Copy link
Copy Markdown
Member

Completely forgot about this feature!

Enable the Copilot CLI file-based OTel exporter for pull request dashboard classifications and print the raw JSONL output into the workflow log. This keeps classification output simple while making Copilot token and usage telemetry available directly from Actions logs for investigation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR simplifies the pull-request-dashboard's Copilot classification path. Instead of requesting JSON output from the copilot CLI and parsing per-call token usage out of the JSONL stream, it now runs the CLI in default (text) mode with --silent, reads the model's answer directly from stdout, and enables the CLI's file-based OpenTelemetry exporter so the raw OTel JSONL is dumped into the workflow log (stderr) for investigation. All of the bespoke usage-normalization and usage-aggregation code is removed.

Changes:

  • Replaced JSONL/usage parsing in run_llm_for_thread with a temp-dir OTel file exporter (COPILOT_OTEL_FILE_EXPORTER_PATH / COPILOT_OTEL_EXPORTER_TYPE=file) and print_copilot_otel_file, reading the response straight from proc.stdout.
  • Switched the CLI invocation from --output-format json to --silent, and dropped the usage field from classification records.
  • Removed the entire Copilot usage aggregation/reporting pipeline from dashboard.py (empty_copilot_usage, add_copilot_usage, copilot_usage_from_results, print_copilot_usage_summary, and the copilot_usage dataclass field).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/scripts/pull-request-dashboard/classification.py Drops usage parsing; enables OTel file exporter, logs raw JSONL, and parses the decision from raw stdout instead of JSONL.
.github/scripts/pull-request-dashboard/dashboard.py Removes the now-unused Copilot usage aggregation/summary logic and the copilot_usage field from DashboardCalculation.

Note: The behavioral switch from --output-format json (parsed JSONL assistant.message content) to default text output with --silent (raw proc.stdout) is the central risk here — correct classification now depends on the CLI still printing the model's JSON answer to stdout in this mode. This couldn't be fully verified from the repository itself, so it warrants human confirmation against the pinned Copilot CLI behavior.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/scripts/pull-request-dashboard/classification.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@trask trask marked this pull request as ready for review June 26, 2026 21:56
@trask trask requested a review from a team as a code owner June 26, 2026 21:56

@maryliag maryliag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@maryliag maryliag merged commit 6e0a4b6 into open-telemetry:main Jun 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants