Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 21 additions & 14 deletions skills/pr-management-code-review/review-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,7 @@ For each finding, record:
metadata coverage across the supported version space, the
compatibility classification (broken, compatible, or unknown),
and either one concrete supported resolution that still fails,
the conflicting paths that make the intersection empty,
an explicit justification that exhaustive evidence contains no
failing resolution, or a statement that partial evidence leaves
compatibility unknown
Expand All @@ -346,20 +347,26 @@ requirement. Build a constraint ledger for the affected package:
enumerate every mandatory direct and transitive path, apply
environment markers, and intersect their ranges with lock or
resolver metadata and the supported-version matrix when present.
Then identify exact versions that satisfy every constraint but
still lack the required API. Record that ledger and resolution in
`dependency_evidence`. A direct lower bound by itself is not a
failing resolution when another mandatory path narrows the range.
If the available evidence does not identify a concrete failing
resolution, the runtime incompatibility claim remains unsubstantiated
and must not be raised. Absence of a failing resolution proves
compatibility only when the inspected metadata exhaustively covers
the supported version space; record what makes that coverage
exhaustive. When coverage is partial and contains no concrete failing
resolution, classify runtime compatibility as unknown. That unknown
state cannot support a runtime incompatibility finding, but it does
not suppress a separate policy finding backed by the adopter's own
dependency or release rules.
If the effective intersection is empty in any supported environment,
classify the dependency graph as broken because it is uninstallable.
Record the conflicting paths and environment in `dependency_evidence`;
an uninstallable graph does not need a concrete failing resolution and
must never be classified as compatible. Otherwise, identify exact
versions that satisfy every constraint but still lack the required API.
Record that ledger and resolution in `dependency_evidence`. A direct
lower bound by itself is not a failing resolution when another
mandatory path narrows the range.
For a non-empty effective intersection, if the available evidence does
not identify a concrete failing resolution, the runtime incompatibility
claim remains unsubstantiated and must not be raised. For that non-empty
intersection, absence of a failing resolution proves compatibility only
when the inspected metadata exhaustively covers the supported version
space; record what makes that coverage exhaustive. When a non-empty
effective intersection has partial coverage and no concrete failing
resolution, classify runtime compatibility as unknown. That unknown state
cannot support a runtime incompatibility finding, but it does not suppress
a separate policy finding backed by the adopter's own dependency or release
rules.

After classifying runtime compatibility and before prescribing any
remediation, read the applicable per-area `AGENTS.md` discovered in
Expand Down
2 changes: 1 addition & 1 deletion tools/skill-evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Suites are currently implemented for:
- **issue-reproducer** — 27 cases across 7 steps (step-1-inventory, step-2-pick-candidate, step-3-classify-shape, step-5.5-confirm, step-7-verify, step-8-baselines, step-10-compose-verdict)
- **issue-fix-workflow** — 12 cases across 4 steps (step-2-locate-area, step-6-scope-check, step-7-compose-commit, step-8-handback)
- **issue-reassess-stats** — 8 cases across 3 steps (step-1-fetch-verdicts, step-2-classify, step-3-aggregate)
- **pr-management-code-review** — 115 cases across 27 suites (selector-resolution, step-1-selectors-match-chips, step-2.5-slop-detection, step-3-security-disclosure-scan, step-3-ai-authorship-disclosure, step-4-* checks, step-5-adversarial-integration, step-6-disposition, step-7b-review-body-attribution, review-risk-classify, injection-guard, review-disposition, review-handoff)
- **pr-management-code-review** — 116 cases across 27 suites (selector-resolution, step-1-selectors-match-chips, step-2.5-slop-detection, step-3-security-disclosure-scan, step-3-ai-authorship-disclosure, step-4-* checks, step-5-adversarial-integration, step-6-disposition, step-7b-review-body-attribution, review-risk-classify, injection-guard, review-disposition, review-handoff)
- **pr-management-mentor** — 20 cases across 2 steps (tone-checks, hand-off)
- **pr-management-stats** — 13 cases across 2 steps (classify, pressure-weight)
- **pr-management-triage** — 33 cases across 3 steps (pre-filter, decision-table, terminal-links)
Expand Down
4 changes: 2 additions & 2 deletions tools/skill-evals/evals/pr-management-code-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

Behavioral evals for the `pr-management-code-review` skill.

## Suites (115 cases total)
## Suites (116 cases total)

| Suite | Step | Cases | What it covers |
|---|---|---|---|
Expand All @@ -23,7 +23,7 @@ Behavioral evals for the `pr-management-code-review` skill.
| step-4-api-correctness | Step 4 | 3 | Breaking public-API change (blocking); optional addition / internal change pass |
| step-4-ai-generated-signals | Step 4 | 3 | Fabricated API, placeholder/stub detection; genuine code passes |
| step-4-code-quality | Step 4 | 3 | Swallowed exception; clean code and linter-handled style nits pass |
| step-4-dependency-compatibility | Step 4 | 5 | Complete constraint ledger; adopter policy selects remediation for compatible, broken, and unknown graphs; environment markers affect supported resolutions |
| step-4-dependency-compatibility | Step 4 | 6 | Complete constraint ledger; adopter policy selects remediation for compatible, broken (including uninstallable), and unknown graphs; environment markers affect supported resolutions |
| step-4-architecture-boundaries | Step 4 | 3 | Lower-layer-imports-higher violation; correct direction / providers→core pass |
| step-4-security-model | Step 4 | 3 | Calibration: vulnerability (blocking) vs known-limitation vs deployment-hardening (no finding) |
| step-4.5-suggested-reviewers | Step 4.5 | 4 | Domain-expert reviewer suggestions from CODEOWNERS + commit history: grounded 2–3 with a committer; empty section when nothing grounds out; prompt-injection resistance (ungrounded body request ignored); exclusion of already-reviewing owners |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@
"pattern": "\"recommended_action\"\\s*:\\s*\"(release_marker|direct_lower_bound)\"",
"flags": "is"
},
"has_required_finding_fields": {
"type": "regex",
"field": "findings",
"pattern": "(?=.*\"file\"\\s*:\\s*\"[^\"]+\")(?=.*\"category\"\\s*:\\s*\"(Code quality|Commits and PRs)\")(?=.*\"severity\"\\s*:\\s*\"(major|minor)\")(?=.*\"runtime_compatibility\"\\s*:\\s*\"(broken|compatible|unknown)\")(?=.*\"mandatory_paths_checked\"\\s*:\\s*[0-9]+)(?=.*\"metadata_coverage\"\\s*:\\s*\"(exhaustive|partial)\")(?=.*\"supported_incompatible_resolution\"\\s*:\\s*(true|false))(?=.*\"reason\"\\s*:\\s*\"[^\"]+\")(?=.*\"suggestion\"\\s*:\\s*\"[^\"]+\")",
"flags": "s"
},
"has_empty_findings": {
"type": "max_length",
"field": "findings",
Expand All @@ -19,6 +25,6 @@
"has_complete_constraint_ledger": {
"type": "judge",
"field": "findings",
"rubric": "Every dependency-version compatibility finding has a non-empty dependency_evidence field that enumerates every mandatory direct and transitive constraint path, states their effective intersection, metadata coverage, and compatibility classification, and provides exactly the applicable conclusion: a concrete supported failing resolution, an exhaustive justification that no failing resolution exists, or a statement that partial evidence leaves compatibility unknown."
"rubric": "Every dependency-version compatibility finding has a non-empty dependency_evidence field that enumerates every mandatory direct and transitive constraint path, states their effective intersection, metadata coverage, and compatibility classification, and provides exactly the applicable conclusion: a concrete supported failing resolution, the conflicting paths that make the intersection empty and the graph uninstallable, an exhaustive justification that no failing resolution exists, or a statement that partial evidence leaves compatibility unknown."
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
"mandatory_paths_checked": 2,
"metadata_coverage": "exhaustive",
"supported_incompatible_resolution": false,
"dependency_evidence": "The direct compat-core>=1.8.0 path and the query-base==1.32.0 transitive compat-core>=1.12.0,<1.15.0 path intersect at >=1.12.0,<1.15.0. The exhaustive supported-version matrix contains 1.12.0, 1.13.0, and 1.14.0; every version exports FeatureUnavailable, so no supported failing resolution exists and runtime compatibility is compatible."
"dependency_evidence": "The direct compat-core>=1.8.0 path and the query-base==1.32.0 transitive compat-core>=1.12.0,<1.15.0 path intersect at >=1.12.0,<1.15.0. The exhaustive supported-version matrix contains 1.12.0, 1.13.0, and 1.14.0; every version exports FeatureUnavailable, so no supported failing resolution exists and runtime compatibility is compatible.",
"suggestion": "Add the exact # use next version comment to the direct compat-core dependency; do not change its lower bound directly."
}
],
"has_dependency_evidence": true,
"has_recommended_action": true,
"has_required_finding_fields": true,
"has_complete_constraint_ledger": true
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
"mandatory_paths_checked": 1,
"metadata_coverage": "partial",
"supported_incompatible_resolution": true,
"dependency_evidence": "The direct compat-core>=1.8.0 requirement is the only mandatory constraint path, so the effective intersection is >=1.8.0. Metadata coverage is partial, but query-base==1.32.0 with compat-core==1.8.0 is a concrete supported resolution that satisfies every declaration and lacks FeatureUnavailable, so runtime compatibility is broken."
"dependency_evidence": "The direct compat-core>=1.8.0 requirement is the only mandatory constraint path, so the effective intersection is >=1.8.0. Metadata coverage is partial, but query-base==1.32.0 with compat-core==1.8.0 is a concrete supported resolution that satisfies every declaration and lacks FeatureUnavailable, so runtime compatibility is broken.",
"suggestion": "Add the exact # use next version comment to the direct compat-core dependency; do not change its lower bound directly."
}
],
"has_dependency_evidence": true,
"has_recommended_action": true,
"has_required_finding_fields": true,
"has_complete_constraint_ledger": true
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
"mandatory_paths_checked": 2,
"metadata_coverage": "partial",
"supported_incompatible_resolution": false,
"dependency_evidence": "The direct compat-core>=1.8.0 path and the inspected query-base==1.32.0 transitive compat-core>=1.12.0 path currently intersect at >=1.12.0. Metadata for later permitted query-base versions is missing, so coverage is partial; no concrete supported failing resolution is demonstrated and runtime compatibility remains unknown."
"dependency_evidence": "The direct compat-core>=1.8.0 path and the inspected query-base==1.32.0 transitive compat-core>=1.12.0 path currently intersect at >=1.12.0. Metadata for later permitted query-base versions is missing, so coverage is partial; no concrete supported failing resolution is demonstrated and runtime compatibility remains unknown.",
"suggestion": "Add the exact # use next version comment to the direct compat-core dependency; do not change its lower bound directly."
}
],
"has_dependency_evidence": true,
"has_recommended_action": true,
"has_required_finding_fields": true,
"has_complete_constraint_ledger": true
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
"mandatory_paths_checked": 2,
"metadata_coverage": "partial",
"supported_incompatible_resolution": true,
"dependency_evidence": "The direct compat-core>=1.8.0 path applies in every supported environment, while the query-base==1.32.0 transitive compat-core>=1.12.0 path applies only when python_version < '3.12'. The effective intersection is >=1.12.0 on Python 3.11 and >=1.8.0 on Python 3.12. Metadata coverage is partial, but Python 3.12 with query-base==1.32.0 and compat-core==1.8.0 is a concrete supported resolution that satisfies every active constraint and lacks FeatureUnavailable, so runtime compatibility is broken."
"dependency_evidence": "The direct compat-core>=1.8.0 path applies in every supported environment, while the query-base==1.32.0 transitive compat-core>=1.12.0 path applies only when python_version < '3.12'. The effective intersection is >=1.12.0 on Python 3.11 and >=1.8.0 on Python 3.12. Metadata coverage is partial, but Python 3.12 with query-base==1.32.0 and compat-core==1.8.0 is a concrete supported resolution that satisfies every active constraint and lacks FeatureUnavailable, so runtime compatibility is broken.",
"suggestion": "Raise the direct compat-core lower bound to >=1.12.0, the first version that exports FeatureUnavailable."
}
],
"has_dependency_evidence": true,
"has_recommended_action": true,
"has_required_finding_fields": true,
"has_complete_constraint_ledger": true
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"findings": [
{
"severity": "major",
"runtime_compatibility": "broken",
"recommended_action": "direct_lower_bound",
"mandatory_paths_checked": 2,
"metadata_coverage": "partial",
"supported_incompatible_resolution": false,
"dependency_evidence": "The direct compat-core>=1.8.0,<2.0.0 path and the query-base==2.0.0 transitive compat-core>=2.0.0 path have an empty effective intersection in every supported environment. Metadata coverage is partial, but the conflicting mandatory paths alone prove that the graph is uninstallable, so runtime compatibility is broken even though no concrete supported resolution exists.",
"suggestion": "Update the direct compat-core requirement to >=2.0.0, the first version that exports FeatureUnavailable, removing the obsolete <2.0.0 cap."
}
],
"has_dependency_evidence": true,
"has_recommended_action": true,
"has_required_finding_fields": true,
"has_complete_constraint_ledger": true
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

Title: Use the next compat-core API in the widget adapter

Diff:

```diff
--- a/packages/widget-adapter/pyproject.toml
+++ b/packages/widget-adapter/pyproject.toml
@@
dependencies = [
"compat-core>=1.8.0,<2.0.0",
"query-base==2.0.0",
]

--- /dev/null
+++ b/packages/widget-adapter/src/widget_adapter/client.py
@@
+from compat_core.exceptions import FeatureUnavailable
+
+def get_client():
+ raise FeatureUnavailable("client support is not installed")
```

Published package metadata:

- `FeatureUnavailable` is first exported by `compat-core` 2.0.0.
- The direct `widget-adapter` requirement creates the mandatory path
`compat-core>=1.8.0,<2.0.0`.
- `query-base` 2.0.0 creates a second mandatory path by declaring
`compat-core>=2.0.0`.
- No `compat-core` version can satisfy both paths, so their effective
intersection is empty in every supported environment.
- The supplied metadata proves the constraint conflict but does not enumerate
the available `compat-core` releases, so metadata coverage is partial.

Repository dependency and release policy:

- Packages are released independently.
- When a direct dependency range conflicts with a mandatory transitive path
and changed code uses an API introduced at that transitive lower bound,
update the direct requirement to that lower bound and remove any obsolete
upper cap.
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"prose_fields": ["dependency_evidence"]
"prose_fields": ["dependency_evidence", "suggestion"]
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ Return ONLY valid JSON with this structure:
Rules:

- Review dependency compatibility only; ignore unrelated categories.
- `runtime_compatibility` is `broken` when a concrete supported resolution
cannot use the changed code, `compatible` when exhaustive evidence rules
out such a resolution, and `unknown` when partial evidence establishes
- `runtime_compatibility` is `broken` when the effective intersection is
empty in any supported environment or when a concrete supported resolution
cannot use the changed code. It is `compatible` when exhaustive evidence
rules out both failures, and `unknown` when partial evidence establishes
neither result.
- `recommended_action` follows the supplied repository policy whether runtime
compatibility is broken, compatible, or unknown. Never assume that broken
Expand All @@ -42,13 +43,14 @@ Rules:
incompatibility even when the remaining coverage is `partial`. Do not call
coverage exhaustive merely because a single counterexample is sufficient.
- `supported_incompatible_resolution` is true only when concrete package
versions can satisfy every mandatory constraint and still lack the API.
versions can satisfy every mandatory constraint and still lack the API. It
is false for an empty intersection because no resolution exists.
- `dependency_evidence` enumerates every mandatory direct and transitive
constraint path, states their effective intersection and metadata coverage,
and records the compatibility classification. It also gives either one
concrete supported failing resolution, an exhaustive justification that no
failing resolution exists, or a statement that partial evidence leaves
compatibility unknown.
concrete supported failing resolution, the conflicting paths that make an
intersection empty, an exhaustive justification that no failing resolution
exists, or a statement that partial evidence leaves compatibility unknown.
- `reason` explains the observed compatibility or policy issue.
- `suggestion` follows the repository policy supplied with the case.
- Unknown runtime compatibility cannot support a runtime incompatibility
Expand Down
Loading