Skip to content

PDX-525: Reconcile step schema against corpus, tier argument rules, fix Provar AI apiId allow-list #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mrdailey99 wants to merge 7 commits into
base: develop
Choose a base branch
from
Open

PDX-525: Reconcile step schema against corpus, tier argument rules, fix Provar AI apiId allow-list #236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5f04704
PDX-525: fix(mcp): reconcile step schema against corpus, tier argumen…
mrdailey99 Jul 20, 2026
83ff387
PDX-525: fix(mcp): close adversarial-review findings on the schema-ti…
mrdailey99 Jul 20, 2026
53034f1
PDX-525: fix(mcp): close re-review findings on diff identity, empty c…
mrdailey99 Jul 20, 2026
cf46d54
PDX-525: fix(mcp): separate diff identity from message, allow-list pa…
mrdailey99 Jul 20, 2026
fbdbcdc
PDX-525: fix(mcp): make value detection class-aware via the canonical…
mrdailey99 Jul 20, 2026
75d6856
PDX-525: fix(mcp): restore Quality Hub parity helper, bound the diff …
mrdailey99 Jul 20, 2026
3597600
PDX-525: fix(mcp): scope aggregate identity per test case, order sche…
mrdailey99 Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ sf provar auth login
claude mcp add provar -s user -- sf provar mcp start --allowed-paths /path/to/your/provar/project
```

📖 **[docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) — full setup, all 42 tools, 6 resources, 11 MCP prompts, troubleshooting.**
📖 **[docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) — full setup, all 43 tools, 6 resources, 11 MCP prompts, troubleshooting.**

---

Expand Down Expand Up @@ -251,7 +251,7 @@ DESCRIPTION
Note: --json is not available on this command — stdout is reserved for MCP traffic.

TOOLS EXPOSED
42 tools across: project inspection & org describe, Page Object and test-case
43 tools across: project inspection & org describe, Page Object and test-case
authoring/validation, test-suite/plan validation, properties files, Quality Hub
(test runs, defects, corpus examples), Provar Automation, ANT build, and NitroX
components. See docs/mcp.md for the full catalogue with schemas and examples.
Expand Down
13 changes: 11 additions & 2 deletions docs/PROVAR_TEST_STEP_REFERENCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,11 +361,20 @@ Valid `action` values: `ObjectHome` | `New` | `View` | `Edit` | `Delete` | `Clon

Performs a single UI interaction on a field or button. The `interaction` URI determines the action type.

**Interaction types:**
**Interaction types (a common subset — the vocabulary is open, NOT limited to these):**

- `ui:interaction?name=action` — click a button or link
- `ui:interaction?name=action` — activate a button or link
- `ui:interaction?name=set` — fill/type into a field or select a picklist value
- `ui:interaction?name=click` — click a control; distinct from `action`, and both are valid
- `ui:interaction?name=file` — upload a file (uses `fileLocation` instead of `value`)
- `ui:interaction?name=check` / `name=uncheck` — tick or untick a checkbox
- `ui:interaction?name=doubleClick`, `name=hover`, `name=invoke`, `name=clear`, `name=toggle`, `name=sfLookup`, `name=sfSelect` — long-tail interactions the IDE recorder emits

> A corpus survey of real Provar projects found **39 distinct interaction names** in
> well-formed `uiInteraction` nodes. Treat the list above as examples, not an
> allow-list. Validation rejects only names borrowed from other frameworks
> (`type`, `fill`, `enter`, `input`, `tap`, `press`): use `set` to type into a field,
> and `action` or `click` to activate a control.

**Locator URI format:** `ui:locator?name=FIELD_OR_BUTTON_NAME&binding=ENCODED_BINDING`

Expand Down
58 changes: 32 additions & 26 deletions docs/VALIDATION_RULE_REGISTRY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,35 +11,38 @@ Provar test-case validation runs in two layers. This registry is the single cano

**The validity bridge (PDX-509):** a `critical` best-practice violation is surfaced into `issues[]` as an `ERROR` and therefore gates `is_valid` — EXCEPT where a Layer-1 check already owns the concept (then it is suppressed to avoid double-reporting). `major`/`minor`/`info` affect `quality_score` (and the `needs_improvement` status) only. The `status` field is tri-state: `invalid` (a critical) / `needs_improvement` (loads but `quality_score < quality_threshold`) / `valid`.

**Counts:** Layer 1 — 23 rules (18 gating). Layer 2 — 179 rules (critical 64 / major 68 / minor 29 / info 18; 58 bridged to `is_valid`).
**Counts:** Layer 1 — 26 rules (19 gating). Layer 2 — 182 rules (critical 64 / major 69 / minor 30 / info 19; 58 bridged to `is_valid`).

## Layer 1 — Structural validity rules

| Rule ID | Severity | Gates is_valid? | Applies to | Checks |
| ------------------------- | -------- | --------------- | ---------- | ---------------------------------------------------------------------------------------------- |
| `TC_001` | ERROR | Yes | document | XML declaration present (<?xml …?> first line). |
| `TC_002` | ERROR | Yes | document | XML is well-formed (parses without error). |
| `TC_003` | ERROR | Yes | document | Root element is <testCase>. |
| `TC_010` | ERROR | Yes | testCase | testCase id, when present, is a non-negative integer (id is optional; guid is the identifier). |
| `TC_011` | ERROR | Yes | testCase | testCase has a guid attribute. |
| `TC_012` | ERROR | Yes | testCase | testCase guid is a valid UUID v4. |
| `TC_020` | ERROR | Yes | testCase | testCase has a <steps> element. |
| `TC_030` | ERROR | Yes | apiCall | Each apiCall has a guid attribute. |
| `TC_031` | ERROR | Yes | apiCall | Each apiCall guid is a valid UUID v4. |
| `TC_032` | ERROR | Yes | apiCall | Each apiCall has an apiId attribute. |
| `TC_033` | WARNING | No | apiCall | Each apiCall has a descriptive name attribute. |
| `TC_034` | ERROR | Yes | apiCall | Each apiCall has a testItemId attribute. |
| `TC_035` | ERROR | Yes | apiCall | apiCall testItemId is a whole number. |
| `DATA-001` | WARNING | No | testCase | <dataTable> only iterates under a test plan; flags direct testCase-mode execution. |
| `VAR-REF-001` | WARNING | No | argument | A whole-token {Var} stored as valueClass="string" (use class="variable"). |
| `VAR-REF-002` | WARNING | No | argument | {Var} tokens embedded in a plain string (use class="compound"). |
| `UI-TARGET-001` | ERROR | Yes | apiCall | UiWithScreen/UiWithRow target uses class="uiTarget". |
| `UI-LOCATOR-001` | ERROR | Yes | apiCall | UI action locator uses class="uiLocator". |
| `UI-INTERACTION-001` | ERROR | Yes | apiCall | UiDoAction interaction uses class="uiInteraction". |
| `UI-ASSERT-STRUCTURE-001` | ERROR | Yes | apiCall | UiAssert uses nested field/column/page assertion containers, not a flat argument. |
| `SETVALUES-STRUCTURE-001` | ERROR | Yes | apiCall | SetValues values argument uses class="valueList" with <namedValues>. |
| `ASSERT-001` | WARNING | No | apiCall | AssertValues namedValues format flagged for variable/Apex comparisons. |
| `COMPARISON-TYPE-001` | ERROR | Yes | apiCall | comparisonType is within the step-scoped enum subset (load-blocking otherwise). |
| Rule ID | Severity | Gates is_valid? | Applies to | Checks |
| ----------------------------- | -------- | --------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `TC_001` | ERROR | Yes | document | XML declaration present (<?xml …?> first line). |
| `TC_002` | ERROR | Yes | document | XML is well-formed (parses without error). |
| `TC_003` | ERROR | Yes | document | Root element is <testCase>. |
| `TC_010` | ERROR | Yes | testCase | testCase id, when present, is a non-negative integer (id is optional; guid is the identifier). |
| `TC_011` | ERROR | Yes | testCase | testCase has a guid attribute. |
| `TC_012` | ERROR | Yes | testCase | testCase guid is a valid UUID v4. |
| `TC_020` | ERROR | Yes | testCase | testCase has a <steps> element. |
| `TC_030` | ERROR | Yes | apiCall | Each apiCall has a guid attribute. |
| `TC_031` | ERROR | Yes | apiCall | Each apiCall guid is a valid UUID v4. |
| `TC_032` | ERROR | Yes | apiCall | Each apiCall has an apiId attribute. |
| `TC_033` | WARNING | No | apiCall | Each apiCall has a descriptive name attribute. |
| `TC_034` | ERROR | Yes | apiCall | Each apiCall has a testItemId attribute. |
| `TC_035` | ERROR | Yes | apiCall | apiCall testItemId is a whole number. |
| `DATA-001` | WARNING | No | testCase | <dataTable> only iterates under a test plan; flags direct testCase-mode execution. |
| `VAR-REF-001` | WARNING | No | argument | A whole-token {Var} stored as valueClass="string" (use class="variable"). |
| `VAR-REF-002` | WARNING | No | argument | {Var} tokens embedded in a plain string (use class="compound"). |
| `UI-TARGET-001` | ERROR | Yes | apiCall | UiWithScreen/UiWithRow target uses class="uiTarget". |
| `UI-LOCATOR-001` | ERROR | Yes | apiCall | UI action locator uses class="uiLocator". |
| `UI-INTERACTION-001` | ERROR | Yes | apiCall | UiDoAction interaction uses class="uiInteraction". |
| `UI-INTERACTION-002` | ERROR | Yes | apiCall | UiDoAction interaction name is a real Provar interaction, not one borrowed from another framework (type/fill/enter/input/tap/press). |
| `UI-ASSERT-STRUCTURE-001` | ERROR | Yes | apiCall | UiAssert field assertions use a nested <uiFieldAssertion>, not a flat argument or a <namedValues> block. |
| `SF-CONNECT-TYPE-001` | WARNING | No | testCase | A Salesforce UI test connects via ApexConnect (API + UI), not a UiConnect with no ApexConnect. |
| `CONNECT-REF-CONSISTENCY-001` | WARNING | No | apiCall | Every connection reference (uiConnectionName/apexConnectionName/…) matches a connect step's resultName. |
| `SETVALUES-STRUCTURE-001` | ERROR | Yes | apiCall | SetValues values argument uses class="valueList" with <namedValues>. |
| `ASSERT-001` | WARNING | No | apiCall | AssertValues namedValues format flagged for variable/Apex comparisons. |
| `COMPARISON-TYPE-001` | ERROR | Yes | apiCall | comparisonType is within the step-scoped enum subset (load-blocking otherwise). |

## Layer 2 — Best-practice rules

Expand Down Expand Up @@ -108,7 +111,10 @@ Provar test-case validation runs in two layers. This registry is the single cano
| `SETVALUES-FUNC-STR-001` | StructureAndGrouping | major | 5 | No | SetValues must not use string interpolation for function calls. |
| `SETVALUES-INVALID-ELEMENT-001` | StructureAndGrouping | critical | 10 | Yes | SetValues must not contain invalid child elements. |
| `SETVALUES-ZERO-IDX-001` | StructureAndGrouping | major | 5 | No | SetValues string expression must not use [0] index. |
| `STEP-IDE-PARITY-001` | StructureAndGrouping | info | 1 | No | Steps should round-trip as IDE-authored XML. |
| `STEP-NAMES-001` | StructureAndGrouping | minor | 2 | No | Custom step names for UI asserts and sets. |
| `STEP-RECOMMENDED-ARGS-001` | StructureAndGrouping | minor | 2 | No | Steps should declare conventionally-expected arguments. |
| `STEP-REQUIRED-ARGS-001` | StructureAndGrouping | major | 5 | No | Steps must declare all schema-required arguments. |
| `STRUCT-GROUP-001` | StructureAndGrouping | minor | 2 | No | All steps are inside Group steps or BDD structure. |
| `STRUCT-SUMMARY-001` | StructureAndGrouping | info | 1 | No | Test case has top-level summary. |
| `UI-ASSERT-STRUCT-001` | StructureAndGrouping | critical | 8 | Yes | UiAssert steps must include all required arguments. |
Expand Down
40 changes: 38 additions & 2 deletions docs/mcp-pilot-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,16 +406,52 @@ NitroX is Provar's Hybrid Model for locators — it maps Salesforce component-ba
**Without an API key configured:**

- `isError` must be `false` (NOT `true`) — the generation workflow must continue
- `examples` must be `[]`
- `source` is present on every path: `corpus`, `bundled`, or `none`
- For a **Salesforce UI** query, `source` is `bundled` and `examples` carries a
known-correct offline example (`quality_tier: "bundled"`), so a brand-new project
still has a validated structural pattern to follow
- For an **unrelated** query (REST, database, NitroX), `source` is `none` and
`examples` is `[]` — the bundled set is deliberately withheld off-topic, because a
mismatched example steers generation worse than no example at all
- `warning` must mention `sf provar auth login`

**On rate limit or network failure:** `source` is `none` and `examples` is `[]` —
bundled examples are withheld so a transient outage is never disguised as "no matches".

**What to look for:**

- The AI acknowledges the missing key and offers to continue without grounding
- The AI acknowledges the missing key and either uses the bundled example or falls
back to `provar_step_schema` for the step types it needs
- No error is thrown that would abort the session

---

### Scenario 10a: Step Schema Lookup Without Corpus Access

**Goal:** Confirm an agent with no API key can still author correct steps, and that the
fallback named in error messages actually resolves to a callable tool.

> "What arguments does a UiWithScreen step require? Then list every UI step type."

**What to look for:**

- The AI calls `provar_step_schema` with `api_id: "UiWithScreen"`, then `category: "UI"`
- Required arguments come back as `uiConnectionName` and `target` — **not** a phantom
`screenName`
- The response separates three tiers: `required_arguments` (Provar fails without
these), `recommended_arguments` (present on 80-99% of real steps), and
`ide_emitted_arguments` (what the Provar IDE writes even when empty)
- The tool is reachable regardless of `PROVAR_MCP_TOOLS` — it is registered in the
`validation`, `qualityhub` and `authoring` groups, because all three name it as a
recovery path

**Also verify the recovery loop end to end:** validate a test case with a structural
error, confirm the error text names `provar_step_schema`, then confirm that tool is
callable in the same session. Guidance naming an unavailable tool is a dead end — the
exact failure this scenario guards against.

---

### Scenario 11: Migration Prompts — Convert a Test from Another Framework

**Goal:** Use the MCP migration prompts to convert a test written in CRT, Selenium, or Playwright into a Provar XML test case in one step.
Expand Down
Loading