Log raw Copilot OTel output#24
Conversation
cf49c8c to
9bd5e2c
Compare
There was a problem hiding this comment.
Pull request overview
This PR simplifies the pull-request-dashboard's Copilot classification path. Instead of requesting JSON output from the copilot CLI and parsing per-call token usage out of the JSONL stream, it now runs the CLI in default (text) mode with --silent, reads the model's answer directly from stdout, and enables the CLI's file-based OpenTelemetry exporter so the raw OTel JSONL is dumped into the workflow log (stderr) for investigation. All of the bespoke usage-normalization and usage-aggregation code is removed.
Changes:
- Replaced JSONL/usage parsing in
run_llm_for_threadwith a temp-dir OTel file exporter (COPILOT_OTEL_FILE_EXPORTER_PATH/COPILOT_OTEL_EXPORTER_TYPE=file) andprint_copilot_otel_file, reading the response straight fromproc.stdout. - Switched the CLI invocation from
--output-format jsonto--silent, and dropped theusagefield from classification records. - Removed the entire Copilot usage aggregation/reporting pipeline from
dashboard.py(empty_copilot_usage,add_copilot_usage,copilot_usage_from_results,print_copilot_usage_summary, and thecopilot_usagedataclass field).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/scripts/pull-request-dashboard/classification.py | Drops usage parsing; enables OTel file exporter, logs raw JSONL, and parses the decision from raw stdout instead of JSONL. |
| .github/scripts/pull-request-dashboard/dashboard.py | Removes the now-unused Copilot usage aggregation/summary logic and the copilot_usage field from DashboardCalculation. |
Note: The behavioral switch from --output-format json (parsed JSONL assistant.message content) to default text output with --silent (raw proc.stdout) is the central risk here — correct classification now depends on the CLI still printing the model's JSON answer to stdout in this mode. This couldn't be fully verified from the repository itself, so it warrants human confirmation against the pinned Copilot CLI behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
9bd5e2c to
d3f1d26
Compare
Completely forgot about this feature!
Enable the Copilot CLI file-based OTel exporter for pull request dashboard classifications and print the raw JSONL output into the workflow log. This keeps classification output simple while making Copilot token and usage telemetry available directly from Actions logs for investigation.