Replace perf-review skill with multi-perspective dd-apm-sdk-review - #12349
Replace perf-review skill with multi-perspective dd-apm-sdk-review#12349robertomonteromiguel wants to merge 15 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11aa8f1af7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Fixes 4 issues flagged in PR review: - Quote/reword the "#3" criterion in the llm-validation suite so it doesn't start a YAML comment and break the parser. - Sync the upstream core SKILL.md fix that grants the conventions and cross-sdk reviewers narrowly scoped read-only access instead of silently degrading to NOT VERIFIED under the read-only restriction. - Sync the upstream core SKILL.md fix that stops the change-set target resolution from silently falling back to origin/master and instead asks for confirmation. - Exempt cached Optional.empty()/OptionalInt.empty()/OptionalLong.empty()/ OptionalDouble.empty() singletons from the J15 allocation rule in the performance override, avoiding guaranteed false positives. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00e16ba8be
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…rides - Stop flagging Config.get()/InstrumenterConfig.get() in the performance override: both just return a static INSTANCE, so the "walks a config-resolution chain" rationale was false and guaranteed a false-positive finding on every hot-path call site. - Move StringIndex from "coming" to "available today" in the performance toolkit list — it already ships in internal-api with of/indexOf/ EmbeddingSupport. - Scope the design override's public-API-surface rule to public/protected members of externally-accessible types, so a package-private addition to a non-exported dd-trace-ot type (OTSpan, OTSpanContext, TypeConverter) doesn't require public-API justification. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collect PR title/labels for reviewers that audit them, treat git diff --no-index exit 1 (found a difference) as success instead of failure in the untracked-file discovery loop, and add the promised "Related skills" section to the report template. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8249ed4b94
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Sync SKILL.md from dd-apm-sdk-review-core with the fix for the untracked-file diff error masking (a real access error was being treated the same as an expected diff and silently swallowed). Also relax three performance addenda (J6, J13, J14) in the dd-trace-java overrides that were producing deterministic false positives: weak-reference strengthening only when the referent is retained past the probe, defensive copies allowed for stable snapshots/ownership transfer, and the capturing-lambda-in- computeIfAbsent case moved to flag-as-measure since escape analysis can eliminate it on a cache hit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fbb5b65637
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Relocate dd-apm-sdk-review-overrides/ from the repo root to .agents/dd-apm-sdk-review-overrides/ (sibling of .agents/skills/, not nested inside the skill folder itself). Adding content under the already-accepted .agents/ directory is lower friction to land than a new top-level folder in this repo's root. Update SKILL.md's fixed relative path (../../../ -> ../../), CODEOWNERS, AGENTS.md, and the .llm-validation/.promptfoo fixtures accordingly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88864c4ef0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- performance.md override: J10 no longer points String.format fixes at datadog.trace.util.Strings, which has no formatting/interpolation helper; recommends concatenation/StringBuilder instead. - .llm-validation: full-review case now includes repo-context.md and the security.md override, which the skill actually requires. - promptfoo suite: replace text-based "not-icontains git diff" asserts (false positive on a report that merely mentions the command) with toolCalls-metadata checks, in both places the pattern occurred. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…core Pulls in the remaining codex review fixes from core: a new Correctness reviewer lens, retry-safe twice-in-a-row no-op, a revert exemption narrowed to prose-only reverts, and human authorization required before pushing when even a degraded review is impossible. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 972011e4ed
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Resolves the PR base via baseRefOid instead of "origin/<baseRefName>" (the latter breaks on a cross-repo PR, since origin is the contributor's fork), and moves secret scanning of untracked/committed content to the point it is first read rather than only right before delegating to reviewers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add correctness.md to the llm-validation instruction/case file lists so the required roster lens is actually exercised. - design.md override: components/ dependency rule now targets external dependencies only, not inter-component deps between bootstrap-safe modules. - performance.md override: benchmark inventory now includes module-level src/jmh source sets, not just the two central benchmark suites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ffa836b7d5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… core Sync from dd-apm-sdk-review-core: hardens the Step 1 untracked-file scanning block (mktemp failure handling, benign-vs-real stderr disambiguation, fail-closed secret pre-filter, expanded secret patterns, stdout-visible SUSPECT marker, leading-dash filename safety) and restores the explicit override-status signal in report-template.md's rule-files section. Addresses unresolved chatgpt-codex findings from dd-trace-js PR #10070; verified via a full 8-lens dd-apm-sdk-review re-run on the fixed diff in dd-trace-js (no P0 remains).
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f7dae6f81
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| BASE=$(git merge-base HEAD "$TARGET" 2>/dev/null) | ||
| if [ -n "$BASE" ]; then | ||
| git diff --stat "$BASE"...HEAD | ||
| git diff "$BASE"...HEAD |
There was a problem hiding this comment.
When a committed change contains a recognizable credential, this command emits the complete value into the tool transcript before the manual inspection at line 129 can redact it. The new pre-scan at lines 99–110 protects only untracked files, so committed, staged, and unstaged secrets remain exposed despite the claim that they receive the same treatment; capture and scan/redact each tracked diff before printing it.
AGENTS.md reference: AGENTS.md:L77-L77
Useful? React with 👍 / 👎.
| - Confidence: flag-with-confidence. | ||
| - Severity: SEV-2/3. | ||
| - Fix: memoize (bounded — see `unbounded-memory`) or compile/compute once and hoist. | ||
| 3. `unbounded-memory` — **Unbounded memory / collection** — a cache/map/collection with no size *and* byte bound, or keyed by a high-cardinality input (per-request data, raw strings, user-supplied dimensions). |
There was a problem hiding this comment.
When a cache has strict count and byte caps but uses request-derived or otherwise high-cardinality keys, the or keyed by a high-cardinality input clause still requires an SEV-1 unbounded-memory finding. This directly conflicts with line 99, which says not to flag a high-cardinality cache merely for existing and to require that it lack a bound, so reviewers can block the mandatory gate for bounded caches that cannot grow without limit; make high cardinality aggravate an actually unbounded structure rather than independently satisfying this check.
AGENTS.md reference: AGENTS.md:L77-L77
Useful? React with 👍 / 👎.
|
|
||
| - **J1 — Escaping allocation defeats Escape Analysis** *(refines universal `per-call-allocation`, `escape-elision-defeated`)*. The JVM scalar-replaces only *non-escaping* short-lived objects. An object stored in the tag map / span / a collection, iterated at serialization, or passed to a virtual/megamorphic call **escapes** → EA can't elide it → real heap allocation. The trap: *"the JIT will scalar-replace it" is false for escaping objects.* **Verify in JFR — EA'd objects don't appear in alloc profiles, so a surviving object in the profile *proves* it escapes.** **Treat a zero-allocation microbenchmark as unverified when the object wraps I/O** (a scope or wrapper spanning a blocking/native call) — C2 cannot inline through a blocking I/O boundary, so a tight-loop benchmark with no real I/O will show scalar-replacement that production won't get; ask for a benchmark that includes realistic I/O before accepting an EA claim for such an object. | ||
| - **J2 — Megamorphic dispatch** *(refines universal `polymorphic-dispatch`)*. **PARKED for PR-review flagging (2026-07-08): do NOT raise megamorphism findings in review yet.** Kept as author-reference + a standing-audit target, not an active review idiom — it's too in-the-weeds to land with most devs, and the rubric needs to bank *legible* wins first (allocation, unbounded memory, regex) to earn trust. A hot call site seeing ≥3 receiver types with no dominant one goes megamorphic; ≤2 types stays bimorphic (inlinable); a dominant receiver (≥90%, `TypeProfileMajorReceiverPercent`) still gets guarded mono-inline. The worst sites accumulate silently across many PRs — a per-PR check only catches a PR that *widens* a site, so this wants a periodic `PrintInlining` census of known hot sites, independent of any single review. | ||
| - **J3 — JNI / native crossing: overhead + virtual-thread pinning** *(refines universal `native-boundary-crossing`)*. JNI call ≈ 100ns–1µs (state transition, arg pin/copy, no inlining across); string args via `GetStringUTFChars` = UTF-16→UTF-8 copy. **A JNI call from a virtual thread pins the carrier** → no other vthreads on that carrier run while pinned → concurrency collapse for vthread-reliant apps. Fix: batch at flush on the writer (platform) thread, keep the app-vthread path pure-Java, transport interned IDs not strings; `@CriticalNative` only for short primitive ops. |
There was a problem hiding this comment.
When the reviewer flags a short primitive JNI call, this rule can recommend @CriticalNative, but that annotation is an Android ART optimization rather than an API available on the HotSpot JDKs targeted by this override; it is absent from this repository and from the installed supported JDKs. The author therefore cannot apply the prescribed remediation in dd-trace-java without introducing an Android-only dependency that would not optimize HotSpot JNI calls, so retain the batching/pure-Java advice and remove or replace this recommendation.
AGENTS.md reference: AGENTS.md:L77-L77
Useful? React with 👍 / 👎.
|
replaced by #12364 |
1 participant
What Does This Do
Adds the
dd-apm-sdk-reviewskill: a multi-perspective review (coherence, security,design, performance, maintainability, conventions, cross-SDK) that runs over a pending
change set and consolidates findings into a single report with an explicit verdict
(BLOCK / APPROVE_WITH_COMMENTS / APPROVE).
Includes:
.agents/skills/dd-apm-sdk-review/), this repo'scontext/overrides (
dd-apm-sdk-review-overrides/)(
.promptfoo/dd-apm-sdk-review/), an equivalent eval suite in a custom format(
.llm-validation/)AGENTS.mdreview-guidelines section making the skilla mandatory push gate
Note: Unlike the equivalent rollout in dd-trace-js, this PR also removes the standalone
perf-reviewskill (.agents/skills/perf-review/,.claude/skills/perf-review), sinceits checks are now folded in as the
performancereviewer perspective insidedd-apm-sdk-review-overrides/reviewers/performance.md.Motivation
We want a consistent, automated review pass — covering correctness, security, and
repo-specific conventions — applied before every push, instead of relying solely on
ad hoc human review. This is phase 1 of rolling that out to dd-trace-java, mirroring
the rollout already done in dd-trace-js (DataDog/dd-trace-js#10070).
Additional Notes
https://github.com/DataDog/dd-apm-sdk-review-core/pull/1
.agents/skills/dd-apm-sdk-review/in this repository.They are exact copies of the files from DataDog/dd-apm-sdk-review-core. Any changes
to them should be made in the upstream repository instead.
dd-apm-sdk-review-overrides/contains the repository- and language-specificreview rules. These are the ones that should be edited directly in this repo.
perf-reviewin favor of the new skill'sperformancereviewerperspective — no standalone performance-review skill remains after this change.
discussion upstream; both eval suites are kept for now, consistent with dd-trace-js.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]