Skip to content

test(nextjs): Add AI provider orchestrion instrumentations to e2e app - #22550

Merged
chargome merged 2 commits into
developfrom
test/nextjs-orchestrion-ai-providers
Jul 23, 2026
Merged

test(nextjs): Add AI provider orchestrion instrumentations to e2e app#22550
chargome merged 2 commits into
developfrom
test/nextjs-orchestrion-ai-providers

Conversation

@chargome

Copy link
Copy Markdown
Member

Adds e2e coverage for openai, anthropic-ai and google-genai orchestrion instrumentations in the nextjs-16-orchestrion app. A shared node:http mock server stands in for the three provider APIs so the real SDK clients emit gen_ai spans without live credentials. These libraries are bundled (internalized) and instrumented by the orchestrion build-time loader.

Closes #22506

Add e2e coverage for openai, anthropic-ai and google-genai orchestrion instrumentations in the
nextjs-16-orchestrion app. A shared node:http mock server stands in for the three provider APIs so
the real SDK clients emit gen_ai spans without live credentials. These libraries are bundled
(internalized) and instrumented by the orchestrion build-time loader.

Fixes #22506
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chargome chargome self-assigned this Jul 23, 2026
@chargome

chargome commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

buglitzer run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit e8be434. Configure here.

Replace the `/\/v1beta\/models\/.+:generateContent$/` match with linear startsWith/endsWith
string checks to resolve the CodeQL js/polynomial-redos alert. Equivalent matching, no backtracking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chargome
chargome marked this pull request as ready for review July 23, 2026 12:15
id: 'msg_mock123',
type: 'message',
model,
role: 'assistant',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The mock AI server returns finishReason: 'stop' in lowercase, which is inconsistent with the real API and existing integration tests that expect the uppercase 'STOP'.
Severity: LOW

Suggested Fix

In ai-mock-server.mjs, change the finishReason value from 'stop' to 'STOP' to match the behavior of the actual Google GenAI API and align with existing test expectations. This change should be applied to all mock responses within the file.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location:
dev-packages/e2e-tests/test-applications/nextjs-16-orchestrion/ai-mock-server.mjs#L58

Potential issue: The mock server introduced for end-to-end tests returns a lowercase
`finishReason: 'stop'`. This is inconsistent with the actual Google GenAI API, which
returns uppercase enum values like `'STOP'`. While the current tests do not assert on
this attribute, existing integration tests expect the uppercase value. This discrepancy
in the test infrastructure could cause future tests that validate `finishReason` to fail
or could mask potential bugs in the instrumentation code if it doesn't handle different
casings correctly.

Did we get this right? 👍 / 👎 to inform future reviews.

@chargome
chargome merged commit f8dc497 into develop Jul 23, 2026
44 checks passed
@chargome
chargome deleted the test/nextjs-orchestrion-ai-providers branch July 23, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI provider coverage

3 participants