From 0d769c26a173f2215c015cfc3345466e7242b25b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 8 Sep 2026 13:45:58 +0000 Subject: [PATCH 01/19] fix(check): flag an enumeration decision with no empty outcome (MDL-WF06) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A workflow `decision` branching on an enumeration passed `mxcli check` and `mxcli exec`, then mxbuild rejected the project with CE6686 "The current outcomes of the decision activity do not match the configured expression. Regenerate the outcomes." Mendix generates a decision's outcome set as one outcome per enumeration value PLUS one with an empty value, and mxbuild compares the stored set against that generated set. Nothing in mxcli knew about the empty outcome: MDL-WF03 validated each outcome NAME and explicitly skipped the empty one, and no rule looked at the set. Measured on mxbuild 11.10.0 in a blank app, and each measurement changed the design: - the two-value decision is 1 error; adding `'' -> { }` takes it to 0. - a decision on an attribute carrying a REQUIRED (not null) validation rule is still CE6686 — the condition is on the enumeration type, not on the value, so this is an error rather than a warning. - a `call microflow` activity branching on an enumeration-returning microflow fails and clears identically, so the rule runs at both call sites. - ALTER WORKFLOW reaches the same error through INSERT AFTER / REPLACE ACTIVITY, where no workflow rule had ever run; MDL-WF06 now validates what an ALTER introduces. The others stay CREATE-only: MDL-WF01/WF02 describe a state a later `SET ACTIVITY` can repair, MDL-WF05 needs activities the statement cannot see. mxbuild wants set equality, so a missing enumeration VALUE is CE6686 too; that half needs the enumeration's definition and belongs to the reference pass, and the code says so. Outcomes are classified with buildConditionOutcome's own switch so check and writer cannot drift. Verified in both directions: the .fail.mdl fixture executed with --no-check gives `mx check` 3 errors, all CE6686, one per rule hit; the identical script with the empty outcomes gives 0 and is the positive control `make check-mdl` requires to pass. Stubbing the rule fails exactly the three positive tests and no others. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01L45JFiJ6y58ftg2zWVEq7h --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../skills/mendix/cheatsheet-errors/SKILL.md | 1 + .../skills/mendix/write-workflows/SKILL.md | 16 ++ CHANGELOG.md | 12 ++ cmd/mxcli/syntax/features_workflow.go | 14 +- docs-wiki/bug-patterns/check-mxbuild-drift.md | 8 + docs/01-project/MDL_QUICK_REFERENCE.md | 17 ++ .../PROPOSAL_check_mxbuild_gap_heuristics.md | 18 ++ .../wf-enum-decision-empty-outcome.fail.mdl | 74 +++++++++ .../wf-enum-decision-empty-outcome.mdl | 70 ++++++++ mdl/executor/validate_program.go | 5 + mdl/executor/validate_workflow.go | 146 ++++++++++++++++ mdl/executor/validate_workflow_test.go | 157 ++++++++++++++++++ 13 files changed, 537 insertions(+), 2 deletions(-) create mode 100644 mdl-examples/bug-tests/wf-enum-decision-empty-outcome.fail.mdl create mode 100644 mdl-examples/bug-tests/wf-enum-decision-empty-outcome.mdl diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 5a2ab9f65b..b810fb205f 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -557,3 +557,4 @@ {"area": "mdl/executor", "date": "2026-09-06", "symptom": "A repeatable widget property written as a property value had two failure modes: `attributes: [(attributeName: 'x')]` (single key) checked CLEAN, exec'd successfully and vanished from storage; `[(k: v, k2: v2)]` (multi key) died as `missing ')' at ','`. Reported upstream as mendixlabs/mxcli#999 against FileUploader allowedFileFormats / customButtons.", "cause": "propertyValueV3's array alternative is a list of EXPRESSIONS. `(k: v)` happens to be a valid expression, so the single-key form parsed and the visitor flattened it to []string{\"(attributeName:'x')\"} — a value no widget writer claims, hence the silent drop. `(k: v, k2: v2)` is not an expression, hence the parse error. The two symptoms had ONE cause and looked unrelated.", "file": "mdl/executor/validate_widget_object_property.go", "insight": "The dangerous half was invisible to the one rule that might have caught it: MDL-WIDGET07 ('not recognized, will be silently dropped') fires only when the property is UNKNOWN, so it warned without a project and stayed correctly silent with the widget definition present — i.e. it went quiet in exactly the real-world case. Measuring a diagnostic without -p and concluding it covers the case is the recurring trap. Fix shape: parse the bad form deliberately so BOTH shapes reach one semantic error, rather than leaving the multi-key one as a cryptic parse failure — the grammar alternative exists only to be rejected, and is ordered before the expression array so the single-key form stops being flattened. Do NOT wire it to the object-list builder: the container form already works, and two spellings for one construct is the anti-pattern the design guide names. Make the message rewrite the author's own entry into the working form, so the error carries its remedy. Corpus diff was 0 of 532 scripts, which is necessary and NOT sufficient — it compares diagnostics and cannot see an ordinary array captured into the wrong AST type, so assert the untouched shapes directly.", "issue": "mendixlabs/mxcli#999"} {"area": "mdl/executor", "date": "2026-09-07", "symptom": "`describe workflow` emitted MDL that `mxcli check` refused: 6 of the 14 workflows in the 9 demo apps in mx-test-projects/ failed describe -> check. Two rules fired: MDL-WF03 on decision outcomes like 'FactoryManagement.ENUM_InvestigationType.Engineering', and MDL-WF05 on `jump to decision1` / `jump to split1`.", "cause": "Two independent defects behind one symptom. (1) wfOutcomeIdentRe required a BARE identifier, but every Workflows$EnumerationValueConditionOutcome in the corpus stores the QUALIFIED form Module.Enum.Value (7 of 7 non-empty) — the rule was written to catch free text like 'Confirmed closed' and rejecting the dot was collateral, so the describer was right and the validator wrong. (2) MDL had a name slot only on `user task`; every other builder did act.Name = act.Caption, while Mendix resolves JumpToActivity.TargetActivity by activity NAME and Studio Pro names activities by type and ordinal (decision1, split1, callMicroflow1, userTask1, waitForNotification1, timer1) with no relation to the caption. The stored name had nowhere to be emitted to.", "file": "mdl/executor/validate_workflow.go, mdl/grammar/domains/MDLWorkflow.g4, mdl/executor/cmd_workflows.go, mdl/executor/cmd_workflows_write.go", "insight": "The second half was NOT a describer bug, which is what it looked like at first: the grammar had no name slot, so the fix ran grammar -> AST -> visitor -> builder -> describer. Two measurements settled the design and neither was guessable from the code. Studio Pro's stored outcome value decided which side of defect 1 to change — changing the describer to emit the last segment would have 'fixed' the check and written a document unlike every real one. And TargetActivity turned out to hold a NAME STRING, not an ID pointer, which is why a lost name degrades to a jump-to-itself and surfaces as CE6681 'not possible to jump to end activities or jump-to activities' — an error naming a different fault entirely. Emit the name only when it is not derivable (name != caption and != sanitizeActivityName(caption); for call activities, != the called document's short name), so mxcli-authored workflows describe unchanged and the clause appears exactly where it carries information. The control is what makes this provable: the pre-fix binary, built from HEAD~1 in a throwaway worktree, reproduces 6/14 failing where the fixed one is 0/14, and the round-tripped document was read back to confirm decision1..3 / split1 / callMicroflow1..6 landed and both jump targets resolve — mx check at 0 errors alone would NOT have shown that, since a workflow with a jump to itself is perfectly valid.", "issue": "ako/mxcli#408"} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "`mxcli check --references` reported EVERY enumeration as missing — `attribute 'CriticalPathStation': enumeration not found: Approval.StationKey` — while `DESCRIBE ENUMERATION` returned its values, `SHOW ENUMERATIONS` listed it, `exec` wrote the attribute and mxbuild built the project at 0 errors (mendixlabs/mxcli#1071). A pure false negative: the only broken thing was the checker.", "cause": "`enumerationExists` (mdl/executor/helpers.go) matched containers directly — `enum.ContainerID == module.ID` — which only holds for an enumeration sitting in the module ROOT; one inside a FOLDER has the folder as its container. Every other command resolves through the container hierarchy (`h.GetModuleName(h.FindModuleID(e.ContainerID))`), so the reference checker was the only one that could not see inside a folder. Fixed by deferring to `findEnumeration`, deleting the duplicate rather than patching the copy.", "file": "`mdl/executor/helpers.go` (enumerationExists); call sites `mdl/executor/validate.go:447` (CREATE ENTITY) and `:660` (ALTER ENTITY ADD ATTRIBUTE); tests `mdl/executor/validate_enum_folder_test.go`; example `mdl-examples/bug-tests/1071-foldered-enum-references.mdl`", "insight": "This is upstream #976 a second time. That fix corrected DROP's container matching and did NOT sweep for the other callers asking the same question, so the identical bug sat in the reference checker for months — and its own test file already spelled out the class (\"SHOW, DESCRIBE and ALTER all use the container hierarchy... DROP was the one command of the four\"). When a fix is 'this command resolved containers wrongly', grep for every other place that resolves the same containers before closing it; the enumeration existed in TWO implementations and only the interactive one was ever exercised. Two measurement notes: the report read as 'enumerations are never resolved' because the reporter's module keeps them in folders, so the discriminator (root vs foldered, same module, same script) had to be built before anything else made sense; and the blast radius was larger than reported — CREATE ENTITY fails identically and the report only showed ALTER, so the test covers both call sites.", "ce": []} +{"area": "mdl/executor", "date": "2026-09-08", "symptom": "A workflow `decision` branching on an enumeration passed `mxcli check` and `mxcli exec`, then mxbuild rejected the project with CE6686 \"The current outcomes of the decision activity do not match the configured expression. Regenerate the outcomes.\" The same on a `call microflow` activity branching on an enumeration-returning microflow (\"...of the call microflow activity do not match the configured microflow\"). Reachable from ALTER WORKFLOW as well: an `INSERT AFTER … decision` without the empty outcome takes a project sitting at 0 errors to 1.", "cause": "Mendix generates a decision's outcome set as one outcome per enumeration value PLUS one with an EMPTY value, and mxbuild compares the stored set against that generated set. Nothing in mxcli knew about the empty outcome: MDL-WF03 validated each outcome NAME and explicitly skipped the empty one (`o.Value == \"\"` continue), and no rule looked at the SET. Studio Pro's own documents carry the extra Workflows$EnumerationValueConditionOutcome with Value ''. And no workflow rule had ever run on an ALTER-introduced activity — ValidateWorkflow is keyed on CreateWorkflowStmt, so every MDL-WF rule was blind to INSERT AFTER / REPLACE ACTIVITY.", "file": "mdl/executor/validate_workflow.go (checkWorkflowEmptyEnumOutcome, MDL-WF06), mdl-examples/bug-tests/wf-enum-decision-empty-outcome{,.fail}.mdl, mdl/executor/validate_program.go (ValidateAlterWorkflow wiring)", "insight": "The severity was the whole question, and intuition had it backwards. 'The empty outcome must be for nullable attributes' is the obvious reading and it is wrong: measured on mxbuild 11.10.0, a decision on an attribute carrying a REQUIRED (not null) validation rule is still CE6686 without `'' -> { }` — so it is an unconditional property of the enumeration TYPE, not of the value, and the rule is an error rather than a warning. Two more measurements shaped the scope and neither was guessable. (1) The same CE fires on a call-microflow activity branching on an enumeration return and clears the same way, so the rule runs at BOTH call sites — the reported bug named only decisions, and stopping there would have left half the class open (the 'probe every sibling' rule). (2) mxbuild wants set EQUALITY, not 'contains empty': a two-value enum with Standard + '' is also 1 error. That half needs the enumeration's definition, so it belongs to the reference pass, not to a syntax-only rule — worth stating in the code so the next person does not read the rule as complete. Classify outcomes with buildConditionOutcome's own switch (True/False -> boolean, Default -> void, everything else -> enum) rather than a second reading of 'looks like an enum value', or the two drift on the first grammar change. The control that proves the rule detects anything is a PAIR of fixtures, not the .fail.mdl alone: `.fail.mdl` only asserts a non-zero exit, which a rule that refused every enumeration decision would also produce — so the identical script WITH the empty outcomes sits next to it and must keep passing (0 errors on mxbuild, measured both ways via exec --no-check). The ALTER half is the 'probe every sibling' rule paying out twice: the reported construct was a CREATE decision, and the same statement shape is reachable through ALTER, where NOTHING was validated. Port only the rules whose verdict is complete in the introduced subtree — MDL-WF06 qualifies, MDL-WF01/WF02 do not (a later `SET ACTIVITY … PAGE` in the same script repairs them) and MDL-WF05 cannot (it resolves jump targets against activities the ALTER statement never sees). The false-positive worry for MDL-WF06 on ALTER — a following `INSERT OUTCOME '' ON decisionN` completing the set — turned out not to exist, and the probe found a separate defect instead: INSERT OUTCOME on a decision writes a Workflows$UserTaskOutcome into a ConditionOutcome list, and the project then fails to LOAD (System.InvalidCastException at UnitContentsLoader.FillProperties), which is the MDL-WF04 class, not a build error.", "refs": ["ako/mxcli#408"], "ce": ["CE6686"], "rules": ["MDL-WF06"]} diff --git a/.claude/skills/mendix/cheatsheet-errors/SKILL.md b/.claude/skills/mendix/cheatsheet-errors/SKILL.md index 153dfc11d3..0bed537717 100644 --- a/.claude/skills/mendix/cheatsheet-errors/SKILL.md +++ b/.claude/skills/mendix/cheatsheet-errors/SKILL.md @@ -268,6 +268,7 @@ Run with `-p` for the fullest coverage. | CE1571 | No argument selected for parameter | `$currentObject` in a control-bar button (not row-scoped) — `check` flags MDL-BUTTON01 | | CE1834 | The 'Page' property is required | Workflow user task without a `page` — `check` flags MDL-WF01 | | CE1876 | Single outcome must not contain activities | Single-outcome user task with a nested activity flow — `check` flags MDL-WF02 | +| CE6686 | Outcomes do not match the configured expression/microflow | An enumeration decision or call-microflow activity missing the empty outcome (`'' -> { }`) or an enumeration value — `check` flags the missing empty one as MDL-WF06 | | CW0094 | Variable never used | Unused parameter/variable | ## Quick Validation Checklist diff --git a/.claude/skills/mendix/write-workflows/SKILL.md b/.claude/skills/mendix/write-workflows/SKILL.md index 8e9a2ce058..654b281abd 100644 --- a/.claude/skills/mendix/write-workflows/SKILL.md +++ b/.claude/skills/mendix/write-workflows/SKILL.md @@ -93,6 +93,13 @@ begin true -> { call microflow Module.ACT_Escalate; } false -> { call microflow Module.ACT_AutoApprove; }; + -- An ENUM decision needs every value PLUS the empty one (see below). + decision decision2 '$WorkflowContext/Kind' + outcomes + 'Module.Kind.Standard' -> { } + 'Module.Kind.Priority' -> { } + '' -> { }; + -- Parallel split: independent branches run concurrently parallel split split1 path 1 { call microflow Module.ACT_Notify; } @@ -280,6 +287,15 @@ documented in `system-module`. task (`CE1834`). Bind the page to `System.WorkflowUserTask`. - A user task / decision with a single outcome and no activity can trip `CE1876` — give each branch a body or a distinct outcome. +- An **enumeration** decision needs an outcome for the **empty value** as well + as one per enumeration value: Mendix generates that set and MxBuild compares + the stored outcomes against it, so anything else is `CE6686` ("Regenerate the + outcomes"). Write `'' -> { }` alongside the named values; `check` reports a + missing one as `MDL-WF06`. The same applies to a `call microflow` activity + branching on an enumeration return, and to a decision introduced by + `ALTER WORKFLOW … INSERT AFTER` / `REPLACE ACTIVITY`. A **required + (`not null`) attribute does not exempt it** — measured, the empty outcome is + still required. Boolean (`true`/`false`) decisions do not take one. - The context **Parameter entity must be persistent**. - Write the context variable as **`$WorkflowContext`**, matching the parameter name exactly. Mendix expressions are case-sensitive on 11.9+, so a lowercase diff --git a/CHANGELOG.md b/CHANGELOG.md index 14393db1ef..73673676cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **An enumeration `decision` passed `check` and `exec`, then failed the build with CE6686** — "The current outcomes of the decision activity do not match the configured expression. Regenerate the outcomes." Mendix generates a decision's outcomes as one per enumeration value **plus one for the empty value**, and mxbuild compares the stored set against that generated set — so an enumeration decision written without `'' -> { }` is a build error mxcli had nothing to say about. `mxcli check` now reports it as **MDL-WF06** (error), which also refuses it at `exec` before anything is written. + + The severity turned on a measurement, not on the reading that suggests itself. "The empty branch must be for attributes that can be empty" is wrong: on mxbuild 11.10.0 a decision on an attribute carrying a **required (`not null`)** validation rule is still CE6686 without it. The condition is on the enumeration type, not on the value, so this is an error rather than a warning. + + It fires on `call microflow` activities too. A call branching on an enumeration-returning microflow produces the same CE6686 ("...of the call microflow activity do not match the configured microflow") and clears the same way — the report named only decisions, and covering just those would have left half the class open. Boolean (`true`/`false`) decisions need no empty outcome and are untouched; outcomes are classified with the writer's own switch rather than a second reading of what looks like an enumeration value. + + mxbuild wants set *equality*, so a missing enumeration **value** is CE6686 as well (measured: `Standard` + `''` on a two-value enum is 1 error). That half needs the enumeration's definition and belongs to the reference pass, so the syntax-only rule reports only the empty outcome. + + `ALTER WORKFLOW` is covered too, and was the second half of the same gap: an `INSERT AFTER … decision` (or `REPLACE ACTIVITY`, or an activity inside an inserted outcome, path, branch or boundary event) reaches the identical CE6686 — measured, inserting one into a project sitting at 0 errors takes it to 1 — and no workflow rule had ever looked at an ALTER-introduced activity. MDL-WF06 now runs over what the statement adds. The other workflow rules deliberately stay CREATE-only: MDL-WF01/WF02 describe a state a later `SET ACTIVITY` in the same script can repair, and MDL-WF05 resolves jump targets against activities an ALTER statement cannot see. + + Verified in both directions on a blank 11.10.0 app: `mdl-examples/bug-tests/wf-enum-decision-empty-outcome.fail.mdl` executed with `--no-check` gives `mx check` 2 errors, both CE6686, and the identical script with the empty outcomes (`wf-enum-decision-empty-outcome.mdl`, the positive control `make check-mdl` requires to pass) gives 0. + - **`check --references` reported an enumeration in a folder as missing** (mendixlabs/mxcli#1071) — `attribute 'CriticalPathStation': enumeration not found: Approval.StationKey`, while `DESCRIBE ENUMERATION` returned its values, `SHOW ENUMERATIONS` listed it, `exec` wrote the attribute and mxbuild built the project at 0 errors. A pure false negative, and it read as *"enumerations are never resolved"* because the reporting project keeps its enumerations in folders. `enumerationExists` matched containers directly — `enum.ContainerID == module.ID` — which only ever holds for an enumeration sitting in the module **root**; one inside a folder has the folder as its container. Every other command resolves through the container hierarchy, which walks folders up to the module, so the reference checker was the only one that could not see inside one. It now defers to `findEnumeration`, deleting the duplicate rather than patching the copy — and picking up the live-over-excluded handling the copy never had. diff --git a/cmd/mxcli/syntax/features_workflow.go b/cmd/mxcli/syntax/features_workflow.go index fc511ae873..25f67653d7 100644 --- a/cmd/mxcli/syntax/features_workflow.go +++ b/cmd/mxcli/syntax/features_workflow.go @@ -74,8 +74,18 @@ func init() { // outcome does not ('OK' { }). The two read alike but are separate // grammar rules, so the arrow is easy to drop — this entry did, and // taught the broken form until TestExamplesParse started checking it. - Syntax: "DECISION [] [''] [COMMENT '']\n OUTCOMES '' -> { } ...;", - Example: "DECISION 'Check amount'\n OUTCOMES\n 'Under 1000' -> { }\n 'Over 1000' -> {\n USER TASK ManagerApproval 'Manager must approve'\n OUTCOMES 'OK' { };\n };", + Syntax: "DECISION [] [''] [COMMENT '']\n OUTCOMES TRUE -> { } FALSE -> { };\n\n" + + "-- branching on an enumeration: one outcome per value PLUS the empty one\n" + + "DECISION [] ['']\n OUTCOMES '' -> { } ... '' -> { };", + // An outcome name is an enumeration value identifier, so free text like + // 'Under 1000' is refused (MDL-WF03) — the example taught that form until + // it was corrected. An enumeration decision also needs the EMPTY outcome: + // Mendix generates one outcome per value plus one for the empty value and + // MxBuild compares against that set (CE6686, MDL-WF06). + Example: "-- boolean condition: exactly TRUE and FALSE, no empty outcome\n" + + "DECISION decision1 'Check amount'\n OUTCOMES\n TRUE -> {\n USER TASK ManagerApproval 'Manager must approve'\n OUTCOMES 'OK' { };\n }\n FALSE -> { };\n\n" + + "-- enumeration: every value, plus '' for the empty value (CE6686)\n" + + "DECISION decision2 '$WorkflowContext/Kind'\n OUTCOMES\n 'Sales.Kind.Standard' -> { }\n 'Sales.Kind.Priority' -> { }\n '' -> { };", SeeAlso: []string{"workflow.create", "workflow.parallel-split"}, }) diff --git a/docs-wiki/bug-patterns/check-mxbuild-drift.md b/docs-wiki/bug-patterns/check-mxbuild-drift.md index 5a0ea65176..bb7a784d75 100644 --- a/docs-wiki/bug-patterns/check-mxbuild-drift.md +++ b/docs-wiki/bug-patterns/check-mxbuild-drift.md @@ -53,6 +53,14 @@ of 374 example files and 3 of the hits were false positives, because the rule read the AST while the outcome depended on what the builder synthesises. The corpus is the cheapest false-positive test available. +**Severity turns on what the builder's condition is actually about, and the +intuitive reading is often the wrong one.** The empty-outcome rule looked like a +warning — surely an enumeration decision only needs an empty branch when the +value can be empty — until a *required* attribute was measured and produced the +same build error. The condition was on the enumeration type, not on the value, +which makes it an error. Measure the exemption you are about to grant; do not +infer it. + **Severity is the design decision, not an afterthought.** A rule whose vocabulary cannot be proven complete — anything about widget properties, or about a name that might be legal in a context the rule cannot see — must be a *warning*, or diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 500023d960..8ab9d8ce90 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -668,6 +668,23 @@ caption, so `describe workflow` emits the name whenever it is not derivable. qualified (`Module.Enum.Approved` — the form Studio Pro stores). Free text with spaces is rejected (`MDL-WF03`). +**An enumeration decision also needs an empty outcome.** Mendix generates one +outcome per enumeration value **plus one for the empty value**, and MxBuild +compares the stored set against that: anything else is CE6686 ("Regenerate the +outcomes"). Write it as `'' -> { }` alongside the named values — `check` reports +a missing one as `MDL-WF06`. It applies to `call microflow` outcomes branching on +an enumeration return as well, and a required (`not null`) attribute does **not** +exempt it. Boolean decisions (`true`/`false`) do not take one. + +```sql + decision '$WorkflowContext/Kind' + outcomes + 'Module.Kind.Standard' -> { } + 'Module.Kind.Priority' -> { } + '' -> { } + ; +``` + **Example:** ```sql create workflow Module.ApprovalFlow diff --git a/docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md b/docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md index 6137ec9988..9d6f9b48f9 100644 --- a/docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md +++ b/docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md @@ -44,6 +44,7 @@ pattern (rule + `.fail.mdl` repro + `fix-issue.md` CE→rule row): | `MDL-WIDGET15` | adjacent inline (Text/Paragraph) dynamictexts fuse | info (layout) | | `MDL031` (pass-through) | view pass-through string column length ≠ source → CE6770 | `--references` | | (assoc validate) | `create association` to/from a view entity → CE6771 | `--references` | +| `MDL-WF06` | enumeration decision / call-microflow outcomes with no empty-valued branch → CE6686 | syntax-only | Two more ledger cases in this class were closed by **fixing the write path** rather than adding a check — the MDL is structurally valid, so `check` can't see it; the @@ -52,6 +53,23 @@ orphaned index left by `create or modify` dropping an indexed attribute (which *crashed* `mx check`). These belong to the same "check ↔ build parity" mission but are writer fixes, not heuristics. +**`MDL-WF06` (2026-09).** Mendix generates a decision's outcomes as *one per +enumeration value plus one for the empty value*, and MxBuild compares the stored +set against that generated set — so an enumeration decision written without +`'' -> { }` is CE6686 ("Regenerate the outcomes"), which `check` and `exec` both +accepted. Measured on mxbuild 11.10.0 in a blank app: the two-value decision is 1 +error and adding the empty outcome takes it to 0; a **required (`not null`)** +attribute does *not* exempt it (still CE6686), which is what settles the severity +as error rather than warning; and a `call microflow` activity branching on an +enumeration return fails and clears identically, so the rule runs at both call +sites. mxbuild wants set *equality*, so a missing enumeration **value** is CE6686 +too — that half needs the enumeration's definition and belongs to the reference +pass, not to a syntax-only rule. It is also the first workflow rule to run on +`ALTER WORKFLOW`: an inserted or replaced activity reaches the same build error +(measured), and nothing had ever validated one. The other workflow rules stay +CREATE-only on purpose — MDL-WF01/WF02 describe a state a later op in the same +script can repair, and MDL-WF05 needs activities the ALTER statement cannot see. + **Standing policy:** when a new missing check is reported, implement it here (or as a write-path fix when the construct is valid MDL). The two remaining originally- cataloged gaps are **case 3 (CE7412)** and the standalone **case 6 warn**. diff --git a/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.fail.mdl b/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.fail.mdl new file mode 100644 index 0000000000..d8d7b797c6 --- /dev/null +++ b/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.fail.mdl @@ -0,0 +1,74 @@ +-- NEGATIVE TEST (.fail.mdl) — EXPECTED to fail `mxcli check`. +-- `make check-mdl` inverts the exit code: an unexpected pass is a regression of +-- the MDL-WF06 rule. The positive control is +-- wf-enum-decision-empty-outcome.mdl — the same two workflows WITH `'' -> { }`, +-- which must keep passing. +-- +-- Symptom: a decision branching on an enumeration passed `mxcli check` and +-- `mxcli exec`, then MxBuild rejected the project with CE6686 "The current +-- outcomes of the decision activity do not match the configured expression. +-- Regenerate the outcomes." ALTER WORKFLOW reaches it too: inserting such a +-- decision into a project sitting at 0 errors takes `mx check` to 1. +-- +-- Cause: Mendix generates a decision's outcomes as one per enumeration value +-- PLUS one with an EMPTY value, and MxBuild compares the stored set against +-- that generated set. Studio Pro's own documents carry the extra +-- Workflows$EnumerationValueConditionOutcome with Value ''. +-- +-- Measured on mxbuild 11.10.0 in a blank 11.10.0 app: each activity below is 1 +-- error, and adding `'' -> { }` takes the same project to 0. A required +-- (`not null`) attribute does NOT exempt it — still CE6686 — which is why the +-- rule is an error rather than a warning. Boolean (true/false) decisions need +-- no empty outcome. + +create module WFEmpty; + +create enumeration WFEmpty.Kind ( Standard, Priority ); + +create persistent entity WFEmpty.Order ( + Kind : enumeration(WFEmpty.Kind) not null error 'required' +); + +create microflow WFEmpty.ACT_Kind ($Order: WFEmpty.Order) +returns enumeration(WFEmpty.Kind) as $Result +begin + declare $Result enumeration(WFEmpty.Kind) = WFEmpty.Kind.Standard; + return $Result; +end; +/ + +-- The attribute is required, and the empty outcome is still mandatory. +create workflow WFEmpty.EnumDecision + parameter $WorkflowContext: WFEmpty.Order +begin + decision decision1 '$WorkflowContext/Kind' + outcomes + 'WFEmpty.Kind.Standard' -> { } + 'WFEmpty.Kind.Priority' -> { } + ; +end workflow; +/ + +-- The same rule for a call microflow branching on an enumeration return: +-- CE6686 "The current outcomes of the call microflow activity do not match the +-- configured microflow." +create workflow WFEmpty.EnumCallMicroflow + parameter $WorkflowContext: WFEmpty.Order +begin + call microflow WFEmpty.ACT_Kind as callMicroflow1 + outcomes + 'WFEmpty.Kind.Standard' -> { } + 'WFEmpty.Kind.Priority' -> { } + ; +end workflow; +/ + +-- ALTER introduces activities the same way a CREATE body does, and reaches the +-- same build error. Measured: this insert against a project at 0 errors gives +-- `mx check` 1 error, CE6686. +alter workflow WFEmpty.EnumDecision insert after decision1 + decision decision9 '$WorkflowContext/Kind' + outcomes + 'WFEmpty.Kind.Standard' -> { } + 'WFEmpty.Kind.Priority' -> { } + ; diff --git a/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.mdl b/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.mdl new file mode 100644 index 0000000000..c74c536ffc --- /dev/null +++ b/mdl-examples/bug-tests/wf-enum-decision-empty-outcome.mdl @@ -0,0 +1,70 @@ +-- POSITIVE CONTROL for wf-enum-decision-empty-outcome.fail.mdl. +-- +-- The same two workflows WITH the empty outcome. `make check-mdl` requires this +-- one to PASS, so MDL-WF06 cannot be satisfied by refusing every enumeration +-- decision — the rule has to distinguish the two files. +-- +-- Verified on mxbuild 11.10.0: exec'd into a blank 11.10.0 app, `mx check` +-- reports 0 errors; dropping the `'' -> { }` branches (the .fail.mdl next to +-- this one) reports CE6686 for each activity. + +create module WFEmptyOK; + +create enumeration WFEmptyOK.Kind ( Standard, Priority ); + +create persistent entity WFEmptyOK.Order ( + Kind : enumeration(WFEmptyOK.Kind) not null error 'required' +); + +create microflow WFEmptyOK.ACT_Kind ($Order: WFEmptyOK.Order) +returns enumeration(WFEmptyOK.Kind) as $Result +begin + declare $Result enumeration(WFEmptyOK.Kind) = WFEmptyOK.Kind.Standard; + return $Result; +end; +/ + +create workflow WFEmptyOK.EnumDecision + parameter $WorkflowContext: WFEmptyOK.Order +begin + decision decision1 '$WorkflowContext/Kind' + outcomes + 'WFEmptyOK.Kind.Standard' -> { } + 'WFEmptyOK.Kind.Priority' -> { } + '' -> { } + ; +end workflow; +/ + +create workflow WFEmptyOK.EnumCallMicroflow + parameter $WorkflowContext: WFEmptyOK.Order +begin + call microflow WFEmptyOK.ACT_Kind as callMicroflow1 + outcomes + 'WFEmptyOK.Kind.Standard' -> { } + 'WFEmptyOK.Kind.Priority' -> { } + '' -> { } + ; +end workflow; +/ + +-- A boolean decision takes no empty outcome (0 errors on mxbuild as written). +create workflow WFEmptyOK.BooleanDecision + parameter $WorkflowContext: WFEmptyOK.Order +begin + decision decision1 '$WorkflowContext/Kind = WFEmptyOK.Kind.Priority' + outcomes + true -> { } + false -> { } + ; +end workflow; +/ + +-- The ALTER control: the same insert carrying the empty outcome. +alter workflow WFEmptyOK.EnumDecision insert after decision1 + decision decision9 '$WorkflowContext/Kind' + outcomes + 'WFEmptyOK.Kind.Standard' -> { } + 'WFEmptyOK.Kind.Priority' -> { } + '' -> { } + ; diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 842baed239..9c5f6d9978 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -81,6 +81,11 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { if wfStmt, ok := stmt.(*ast.CreateWorkflowStmt); ok { violations = append(violations, ValidateWorkflow(wfStmt)...) } + // An ALTER that inserts or replaces an activity reaches the same build + // errors as a CREATE body; MDL-WF06 is checked over what it introduces. + if altWfStmt, ok := stmt.(*ast.AlterWorkflowStmt); ok { + violations = append(violations, ValidateAlterWorkflow(altWfStmt)...) + } // Check GRANT for member rights Mendix cannot store if grantStmt, ok := stmt.(*ast.GrantEntityAccessStmt); ok { violations = append(violations, ValidateGrantEntityAccess(grantStmt)...) diff --git a/mdl/executor/validate_workflow.go b/mdl/executor/validate_workflow.go index bad0aa3bed..b1f28a6b39 100644 --- a/mdl/executor/validate_workflow.go +++ b/mdl/executor/validate_workflow.go @@ -36,6 +36,7 @@ var wfOutcomeIdentRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A // - MDL-WF02: single-outcome user task containing nested activities (CE1876) // - MDL-WF03: decision / call-microflow outcome that is not a valid // enumeration value identifier +// - MDL-WF06: enumeration outcomes with no empty-valued branch (CE6686) // - MDL-WF04: standalone `annotation` in a workflow body (unloadable model) // - MDL-WF05: `jump to` a target that names no activity (see validate_workflow_jump.go) func ValidateWorkflow(stmt *ast.CreateWorkflowStmt) []linter.Violation { @@ -71,8 +72,10 @@ func ValidateWorkflow(stmt *ast.CreateWorkflowStmt) []linter.Violation { } case *ast.WorkflowDecisionNode: out = append(out, checkWorkflowOutcomeNames(n.Outcomes, "decision", loc)...) + out = append(out, checkWorkflowEmptyEnumOutcome(n.Outcomes, "decision", workflowDecisionLabel(n), loc)...) case *ast.WorkflowCallMicroflowNode: out = append(out, checkWorkflowOutcomeNames(n.Outcomes, "call microflow", loc)...) + out = append(out, checkWorkflowEmptyEnumOutcome(n.Outcomes, "call microflow", workflowCallMicroflowLabel(n), loc)...) case *ast.WorkflowAnnotationActivityNode: // MDL-WF04 — a standalone annotation is written into the workflow's // activity flow, but Mendix constructs every child of that list with a @@ -92,6 +95,58 @@ func ValidateWorkflow(stmt *ast.CreateWorkflowStmt) []linter.Violation { return out } +// ValidateAlterWorkflow applies the outcome-set rule to the activities an ALTER +// WORKFLOW statement introduces (MDL-WF06). +// +// ALTER reaches the same build error as CREATE: an `INSERT AFTER … decision` +// whose outcomes name enumeration values but no empty one is CE6686, measured on +// mxbuild 11.10.0 against a project that was at 0 errors before the ALTER. The +// activity-shaped ops carry an ordinary WorkflowActivityNode, so the check is +// the CREATE one over the introduced subtree — the same shape +// validateAlterWorkflowRefs uses for references. +// +// Only MDL-WF06 runs here. The others are not simply un-ported: MDL-WF01/WF02 +// describe a state a later op in the same script can still repair (`SET ACTIVITY +// … PAGE`), and MDL-WF05 resolves jump targets against activities the statement +// cannot see. The outcome set of an inserted activity is complete where it is +// written — `INSERT OUTCOME` cannot extend it, since on a decision it writes a +// UserTaskOutcome into a ConditionOutcome list and yields a model Mendix cannot +// load at all. +func ValidateAlterWorkflow(stmt *ast.AlterWorkflowStmt) []linter.Violation { + loc := linter.Location{ + Module: stmt.Name.Module, + DocumentType: "workflow", + DocumentName: stmt.Name.Name, + } + var added []ast.WorkflowActivityNode + for _, op := range stmt.Operations { + switch o := op.(type) { + case *ast.InsertAfterOp: + added = append(added, o.NewActivity) + case *ast.ReplaceActivityOp: + added = append(added, o.NewActivity) + case *ast.InsertOutcomeOp: + added = append(added, o.Activities...) + case *ast.InsertPathOp: + added = append(added, o.Activities...) + case *ast.InsertBranchOp: + added = append(added, o.Activities...) + case *ast.InsertBoundaryEventOp: + added = append(added, o.Activities...) + } + } + var out []linter.Violation + walkWorkflowActivities(added, func(a ast.WorkflowActivityNode) { + switch n := a.(type) { + case *ast.WorkflowDecisionNode: + out = append(out, checkWorkflowEmptyEnumOutcome(n.Outcomes, "decision", workflowDecisionLabel(n), loc)...) + case *ast.WorkflowCallMicroflowNode: + out = append(out, checkWorkflowEmptyEnumOutcome(n.Outcomes, "call microflow", workflowCallMicroflowLabel(n), loc)...) + } + }) + return out +} + // checkWorkflowOutcomeNames flags condition-outcome values (decision / call // microflow branches) that are not valid enumeration value identifiers (MDL-WF03). func checkWorkflowOutcomeNames(outcomes []ast.WorkflowConditionOutcomeNode, kind string, loc linter.Location) []linter.Violation { @@ -111,6 +166,97 @@ func checkWorkflowOutcomeNames(outcomes []ast.WorkflowConditionOutcomeNode, kind return out } +// checkWorkflowEmptyEnumOutcome flags an activity that branches on an +// enumeration without an outcome for the EMPTY value (MDL-WF06). +// +// Mendix generates one outcome per enumeration value **plus one with an empty +// value**, and mxbuild compares the stored set against that generated set: +// anything else is CE6686 ("The current outcomes of the ... do not match the +// configured expression/microflow. Regenerate the outcomes."). Studio Pro's own +// documents agree — every enum decision in the FactoryManagement demo app +// stores the extra Workflows$EnumerationValueConditionOutcome with Value ”. +// +// Measured on mxbuild 11.10.0, in a blank 11.10.0 app: +// +// - decision on `$WorkflowContext/Kind` with the two enum values → 1 error, +// CE6686; adding `” -> { }` → 0 errors. +// - the same decision on an attribute carrying a REQUIRED (not null) +// validation rule → still CE6686. The empty outcome is not about whether +// the value can be empty in practice, which is why this is an error rather +// than a warning. +// - a call-microflow activity branching on an enumeration-returning microflow +// → the same CE6686 ("...of the call microflow activity do not match the +// configured microflow"), cleared the same way. Hence both call sites. +// - a boolean decision (true/false) is 0 errors with no empty outcome, so the +// rule must classify outcomes exactly as buildConditionOutcome does. +// +// mxbuild wants set EQUALITY, so a missing enumeration *value* is CE6686 too +// (measured: Standard + ” on a two-value enum is 1 error). That half needs the +// enumeration's definition and so belongs to the reference pass, not here; this +// rule reports only what is decidable from the statement alone. +func checkWorkflowEmptyEnumOutcome(outcomes []ast.WorkflowConditionOutcomeNode, kind, label string, loc linter.Location) []linter.Violation { + var enumValues []string + for _, o := range outcomes { + switch o.Value { + case "True", "False": + // A boolean branch: buildConditionOutcome emits a + // BooleanConditionOutcome and mxbuild wants exactly true/false. + return nil + case "Default": + // A VoidConditionOutcome — not an enumeration branch. + continue + case "": + // The empty-valued enumeration outcome this rule is about. + return nil + default: + enumValues = append(enumValues, o.Value) + } + } + if len(enumValues) == 0 { + return nil + } + // Quote the CE6686 text MxBuild actually prints for this activity kind, so + // searching the build output for it lands here. + ceText := "the current outcomes of the decision activity do not match the configured expression" + if kind == "call microflow" { + ceText = "the current outcomes of the call microflow activity do not match the configured microflow" + } + return []linter.Violation{{ + RuleID: "MDL-WF06", + Severity: linter.SeverityError, + Location: loc, + Message: fmt.Sprintf( + "%s %s branches on an enumeration but has no outcome for the empty value — MxBuild rejects this (CE6686 %q)", + kind, label, ceText), + Suggestion: "Add an empty outcome alongside the named values: `'' -> { }`. Mendix generates one outcome per enumeration value plus one for the empty value, and the stored set must match — a required (not null) attribute does not exempt it.", + }} +} + +// workflowDecisionLabel returns a human-readable label for a decision. +func workflowDecisionLabel(n *ast.WorkflowDecisionNode) string { + switch { + case n.Name != "": + return "'" + n.Name + "'" + case n.Caption != "": + return "'" + n.Caption + "'" + case n.Expression != "": + return "on '" + n.Expression + "'" + } + return "(unnamed)" +} + +// workflowCallMicroflowLabel returns a human-readable label for a call-microflow +// activity. +func workflowCallMicroflowLabel(n *ast.WorkflowCallMicroflowNode) string { + if n.Name != "" { + return "'" + n.Name + "'" + } + if qn := n.Microflow.String(); qn != "" && qn != "." { + return "'" + qn + "'" + } + return "(unnamed)" +} + // workflowUserTaskLabel returns a human-readable label for a user task. func workflowUserTaskLabel(n *ast.WorkflowUserTaskNode) string { if n.Name != "" { diff --git a/mdl/executor/validate_workflow_test.go b/mdl/executor/validate_workflow_test.go index ec31b4fcdb..42ac708bc9 100644 --- a/mdl/executor/validate_workflow_test.go +++ b/mdl/executor/validate_workflow_test.go @@ -238,3 +238,160 @@ end workflow;` t.Fatalf("jump to a named decision/split must resolve, got %v", vs) } } + +// MDL-WF06 — an enumeration decision whose outcomes omit the empty value. +// +// Measured on mxbuild 11.10.0: the decision below is 1 error, CE6686 ("The +// current outcomes of the decision activity do not match the configured +// expression. Regenerate the outcomes."), and adding `” -> { }` takes the same +// project to 0. See TestValidateWorkflow_EnumDecisionWithEmptyOutcomeClean for +// the control. +func TestValidateWorkflow_EnumDecisionWithoutEmptyOutcome(t *testing.T) { + src := wfPreamble + `create workflow WF.W parameter $Ctx: WF.Ctx +begin + decision '$Ctx/Kind' + outcomes + 'WF.Kind.Standard' -> { } + 'WF.Kind.Priority' -> { } + ; +end workflow;` + vs := workflowViolations(t, src) + if !hasRule(vs, "MDL-WF06") { + t.Fatalf("expected MDL-WF06 for an enum decision with no empty outcome, got %v", vs) + } + for _, v := range vs { + if v[0] == "MDL-WF06" && !strings.Contains(v[1], "CE6686") { + t.Errorf("MDL-WF06 should name CE6686, got %q", v[1]) + } + } +} + +// The control: the same decision with the empty outcome is clean. Without this +// the rule could be "always fires on a decision" and the test above would still +// pass. +func TestValidateWorkflow_EnumDecisionWithEmptyOutcomeClean(t *testing.T) { + src := wfPreamble + `create workflow WF.W parameter $Ctx: WF.Ctx +begin + decision '$Ctx/Kind' + outcomes + 'WF.Kind.Standard' -> { } + 'WF.Kind.Priority' -> { } + '' -> { } + ; +end workflow;` + if vs := workflowViolations(t, src); hasRule(vs, "MDL-WF06") { + t.Fatalf("an enum decision carrying the empty outcome must not trigger MDL-WF06, got %v", vs) + } +} + +// A boolean decision needs no empty outcome — mxbuild accepts true/false alone +// (0 errors, measured) — so MDL-WF06 must not fire on one. +func TestValidateWorkflow_BooleanDecisionNoWF06(t *testing.T) { + src := wfPreamble + `create workflow WF.W parameter $Ctx: WF.Ctx +begin + decision '$Ctx/Total > 1000' + outcomes + true -> { } + false -> { } + ; +end workflow;` + if vs := workflowViolations(t, src); hasRule(vs, "MDL-WF06") { + t.Fatalf("boolean decision must not trigger MDL-WF06, got %v", vs) + } +} + +// A call-microflow activity branching on an enumeration return needs the empty +// outcome for the same reason: measured 1 error, CE6686 ("The current outcomes +// of the call microflow activity do not match the configured microflow"), and 0 +// with `” -> { }`. +func TestValidateWorkflow_CallMicroflowEnumOutcomesWithoutEmpty(t *testing.T) { + src := wfPreamble + `create microflow WF.ACT ($Ctx: WF.Ctx) begin return; end; +create workflow WF.W parameter $Ctx: WF.Ctx +begin + call microflow WF.ACT as callMicroflow1 + outcomes + 'WF.Kind.Standard' -> { } + 'WF.Kind.Priority' -> { } + ; +end workflow;` + vs := workflowViolations(t, src) + if !hasRule(vs, "MDL-WF06") { + t.Fatalf("expected MDL-WF06 for enum call-microflow outcomes with no empty outcome, got %v", vs) + } + for _, v := range vs { + if v[0] == "MDL-WF06" && !strings.Contains(v[1], "call microflow") { + t.Errorf("MDL-WF06 should name the activity kind, got %q", v[1]) + } + } +} + +// A void call-microflow activity carries a single DEFAULT outcome, which is not +// an enumeration branch — MDL-WF06 must leave it alone. +func TestValidateWorkflow_CallMicroflowDefaultOutcomeNoWF06(t *testing.T) { + src := wfPreamble + `create microflow WF.ACT ($Ctx: WF.Ctx) begin return; end; +create workflow WF.W parameter $Ctx: WF.Ctx +begin + call microflow WF.ACT as callMicroflow1; +end workflow;` + if vs := workflowViolations(t, src); hasRule(vs, "MDL-WF06") { + t.Fatalf("a call microflow with no explicit outcomes must not trigger MDL-WF06, got %v", vs) + } +} + +// alterWorkflowViolations parses MDL and runs ValidateAlterWorkflow on every +// ALTER WORKFLOW statement. +func alterWorkflowViolations(t *testing.T, src string) [][2]string { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + var out [][2]string + for _, stmt := range prog.Statements { + if wf, ok := stmt.(*ast.AlterWorkflowStmt); ok { + for _, v := range ValidateAlterWorkflow(wf) { + out = append(out, [2]string{v.RuleID, v.Message}) + } + } + } + return out +} + +// MDL-WF06 reaches ALTER as well as CREATE. Measured: inserting this decision +// into a project that was at 0 errors takes `mx check` to 1, CE6686 — and +// nothing looked at an ALTER-introduced activity. +func TestValidateAlterWorkflow_InsertedEnumDecisionWithoutEmptyOutcome(t *testing.T) { + src := `alter workflow WF.W insert after decision1 + decision decision9 '$WorkflowContext/Kind' + outcomes + 'WF.Kind.Standard' -> { } + 'WF.Kind.Priority' -> { } + ;` + if vs := alterWorkflowViolations(t, src); !hasRule(vs, "MDL-WF06") { + t.Fatalf("expected MDL-WF06 for an ALTER-inserted enum decision, got %v", vs) + } +} + +// The control: the same insert carrying the empty outcome is clean. +func TestValidateAlterWorkflow_InsertedEnumDecisionWithEmptyOutcomeClean(t *testing.T) { + src := `alter workflow WF.W insert after decision1 + decision decision9 '$WorkflowContext/Kind' + outcomes + 'WF.Kind.Standard' -> { } + 'WF.Kind.Priority' -> { } + '' -> { } + ;` + if vs := alterWorkflowViolations(t, src); hasRule(vs, "MDL-WF06") { + t.Fatalf("an ALTER-inserted enum decision with the empty outcome must be clean, got %v", vs) + } +} + +// An ALTER that only removes or renames introduces no activity and must stay +// silent — the rule reads what the statement adds, not the stored workflow. +func TestValidateAlterWorkflow_NonInsertingOpsNoWF06(t *testing.T) { + src := `alter workflow WF.W drop activity decision9; +alter workflow WF.W set display 'Approval';` + if vs := alterWorkflowViolations(t, src); len(vs) > 0 { + t.Fatalf("non-inserting ALTER ops must produce no violations, got %v", vs) + } +} From 75acb68b0490d6afe1b689dafd415be9a596d7d9 Mon Sep 17 00:00:00 2001 From: Ako Date: Wed, 9 Sep 2026 05:17:11 +0000 Subject: [PATCH 02/19] feat(catalog): index offline sync configs, and make a synced entity a reference target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 of PROPOSAL_offline_sync_configuration.md. navigation_profiles.OfflineEntityCount said HOW MANY entities a profile syncs and nothing else, so "which entities, and how?" — the first question anyone auditing an offline app asks — needed the document. A count is a summary that cannot be drilled into, not a projection of the data. CATALOG.OFFLINE_ENTITY_CONFIGS now holds a row per configured entity, with the mode, the XPath constraint and CompatibilityMode. The flag is indexed although MDL cannot author it: the catalog reports what is STORED, and a flag invisible to every query is one nobody discovers until it matters. A configured entity also emits a `sync` edge into CATALOG.REFS, so `show references to Sales.Order` names the profiles that download it. That was unanswerable while the same question about a page or a Java action was one query — and it is the question an offline change asks, because changing an entity a profile syncs changes what every device already holds. EVERY mode gets an edge, including the ones that download nothing. The narrowing to ALL and Constrained is tempting and wrong: a profile with `sync X never` still NAMES X, so renaming or dropping it leaves the config dangling, which is precisely what the edge exists to reveal. The test names the quiet modes one by one, so a future narrowing has to delete an assertion that says why rather than watch a total change. moduleOf was a closure in builder_graph.go and is now a shared helper rather than a second copy. The graph views derive a module as everything before the FIRST dot; a variant taking the last dot would silently regroup every node. Verified on ako/TestApp: all seven configs indexed across all six modes, and `show references to Pages.Bus` lists the TabletOffline profile beside the existing create/datasource/parameter edges. TestTables_CoversAllViews caught a real omission: a view added to tables.go is queryable but absent from SHOW CATALOG TABLES until Catalog.Tables() lists it too. Recorded as a finding — two hand-maintained lists of the same thing, and the guard is the only reason it was a five-second fix. Co-Authored-By: Claude Opus 5 --- .../skills/fix-issue/findings/mdl-other.jsonl | 1 + .../skills/mendix/manage-navigation/SKILL.md | 18 ++ CLAUDE.md | 2 +- docs-site/src/tools/catalog-tables.md | 35 ++++ mdl/catalog/builder_graph.go | 7 - mdl/catalog/builder_navigation.go | 46 +++++ mdl/catalog/builder_offline_sync_test.go | 171 ++++++++++++++++++ mdl/catalog/builder_references.go | 25 +++ mdl/catalog/catalog.go | 1 + mdl/catalog/tables.go | 24 +++ 10 files changed, 322 insertions(+), 8 deletions(-) create mode 100644 mdl/catalog/builder_offline_sync_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-other.jsonl b/.claude/skills/fix-issue/findings/mdl-other.jsonl index 93bf4c6122..6cfc09dccc 100644 --- a/.claude/skills/fix-issue/findings/mdl-other.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-other.jsonl @@ -60,3 +60,4 @@ {"area": "mdl/linter", "date": "2026-08-31", "symptom": "`mxcli lint` reports a nanoflow or a rule as a microflow — \"Microflow 'Rule1' has no activities [MPR002]\" about a rule, \"Microflow 'Nanoflow' …\" about a nanoflow. The same wrong noun reaches the JSON and SARIF `documentType` field, where it is not merely cosmetic", "cause": "Microflows, nanoflows and rules share one catalog table (`microflows`, discriminated by `MicroflowType`; the `nanoflows` and `rules` views are filters over it), so `LintContext.Microflows()` yields all three. Eleven rule call sites hardcoded `DocumentType: \"microflow\"`, four of them also putting the word in the message", "file": "`mdl/linter/context.go` (`Microflow.DocumentNoun`/`DocumentNounTitle`), then the call sites in `mdl/linter/rules/`: `empty.go` (MPR002), `conv_loop_commit.go`, `conv_error_handling.go`, `conv_split_caption.go`, `flow_irreducible_graph.go`, `mpr008_overlapping_activities.go`, `mpr011_loop_child_containment.go`, `naming.go`, `validation_feedback.go`", "insight": "**A shared table means a shared iterator, and an iterator yielding three doctypes needs the noun derived, never literal.** Grep `ctx.Microflows()` before assuming a rule is microflow-only — nine rules use it. Fix the labelling only; which documents a rule applies to is a separate question from what the report calls them. Two traps: widening a format string is **not** a compile error (`fmt.Sprintf` is variadic), so `go vet` is the gate that catches the missing argument, not `go build` — it caught two here. And the pre-existing fixtures in `empty_test.go` spell the type title-case (`\"Microflow\"`) where the catalog writes uppercase (`\"MICROFLOW\"`, `mdl/catalog/builder_microflows.go`); harmless while nothing read the column, but a fixture with the wrong case now exercises the fallback instead of the mapping. Control: revert the two lines in `empty.go` and `TestEmptyMicroflowRule_NamesTheDocumentType` reports the symptom verbatim. `mx check` and the build are irrelevant — this is mxcli's own output, not the model", "rules": ["MPR002"]} {"area": "mdl/linter", "date": "2026-09-07", "symptom": "`mxcli lint` CONV010 reported \"ACT_ microflow 'X' contains 'ExclusiveMerge' activity\" on every ACT_ microflow containing an `if` — 122 times on one project, and on a minimal microflow whose ONLY violation was the merge (ako/CapTrackV4 R11).", "cause": "ALLOWED_ACTIVITY_TYPES in conv010_act_microflow_content.star listed ExclusiveSplit and not ExclusiveMerge. An `if` emits both, so the rule permitted the branch and flagged the join it necessarily creates. Fixed by adding ExclusiveMerge, with a test that asks the catalog's own labeller (getMicroflowObjectType) for the two names rather than hardcoding them.", "file": "`.claude/lint-rules/conv010_act_microflow_content.star`; tests `mdl/catalog/lint_rule_vocabulary_test.go`", "insight": "The codebase had already answered this in the other direction and the rule disagreed with it: countMicroflowActivities in mdl/catalog excludes ExclusiveMerge as structural, with a comment saying so. When a lint rule's vocabulary looks wrong, check whether another part of the same package has already classified the same thing. The control matters as much as the fix — LoopedActivity and InheritanceSplit must stay flagged, or a 'widen the list' fix passes the new test and guts the rule. Note also that `mxcli lint` loads Starlark rules ONLY from the project's own `.claude/lint-rules/`, never the embed, so a fixed built-in reaches an existing project only when it re-runs `mxcli init` — which is why a project can keep reporting a rule bug that is already fixed upstream."} {"area": "mdl/catalog", "date": "2026-09-07", "symptom": "A microflow wired as the project's AfterStartupMicroflow reported \"no callers found\" and \"no references found\", and `mxcli lint` QUAL004 said \"is not called from anywhere. Remove if unused.\" Dropping it left a dangling name that `mx check` also missed; only the runtime refused to start (ako/CapTrackV4 049, R13).", "cause": "The runtime calls these, so nothing in the model does, and CATALOG.REFS had no edge kind for a project setting. Added RefKindSettings and extractProjectSettingsRefs, emitting one `settings` edge per setting that names a microflow (AfterStartupMicroflow, BeforeShutdownMicroflow, HealthCheckMicroflow), with the SETTING as the edge's source; added \"settings\" to QUAL004's MICROFLOW_ENTRY_KINDS.", "file": "`mdl/catalog/builder_references.go` (RefKindSettings, extractProjectSettingsRefs, projectSettingsMicroflowRefs); `.claude/lint-rules/orphaned_elements.star`; tests `mdl/catalog/lint_rule_vocabulary_test.go`", "insight": "Same class as the scheduled-event edge, and the rule's own comment had already named the failure mode — \"a kind missing here turns a live document into a false 'not called from anywhere' finding\" — which makes the list of entry kinds worth auditing whenever a new way to invoke a microflow is added. The dangerous half is not the missing reference but the lint rule built on it: QUAL004 does not merely fail to notice, it actively advises deleting the microflow whose deletion breaks the build. The settings list is a literal rather than reflection over ModelSettings, because most of that struct is strings that are not microflow names and a wrong entry would invent an edge rather than miss one."} +{"area": "mdl/catalog", "date": "2026-09-09", "symptom": "A new CATALOG view is queryable by name but absent from `show catalog tables`, so nobody can discover it — caught by TestTables_CoversAllViews, not by any query", "cause": "Adding a view to tables.go creates it in SQLite, but Catalog.Tables() is a separate hand-maintained list and is what SHOW CATALOG TABLES prints. The two are not derived from each other", "file": "`mdl/catalog/catalog.go` (Tables), `mdl/catalog/tables.go`", "insight": "Two hand-maintained lists of the same thing, with a guard test comparing them — the guard is the only reason this is a five-second fix instead of a view nobody finds for months. When adding a catalog view, expect to touch both. The general shape recurs in this codebase (stmtCreateInfo vs projectNameSets.setFor under-reported conflicts for the same reason), so the question to ask of any list is what compares it to its twin", "refs": ["ako/mxcli#413"]} diff --git a/.claude/skills/mendix/manage-navigation/SKILL.md b/.claude/skills/mendix/manage-navigation/SKILL.md index 5103d2657b..79c713bce1 100644 --- a/.claude/skills/mendix/manage-navigation/SKILL.md +++ b/.claude/skills/mendix/manage-navigation/SKILL.md @@ -242,6 +242,24 @@ form. This is the general problem tracked as `mendixlabs/mxcli#750`. **The block replaces the stored list**, the way `menu (...)` replaces the menu. Omitting it leaves the stored configuration alone. +**Ask the catalog which entities sync, rather than reading the profile.** + +```sql +select EntityQualifiedName, SyncMode, XPathConstraint + from CATALOG.OFFLINE_ENTITY_CONFIGS where ProfileName = 'PhoneOffline'; +``` + +And before changing an entity, ask which profiles download it — an offline +change reaches every device that already synced: + +``` +show references to MyModule.Order +``` + +The `sync` row names the profile. Every mode produces one, **including the +modes that download nothing**: a profile with `sync X never` still names `X`, +so renaming or dropping it leaves the configuration dangling. + **Compatibility mode has no syntax.** mxcli reads it, preserves it across a rewrite, and `describe navigation` flags any entity that has it on — it is never silently dropped. diff --git a/CLAUDE.md b/CLAUDE.md index dc8c5c5677..e291e9b03d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -815,7 +815,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati ## Current Implementation Status **Implemented:** -- Offline synchronization (`CREATE NAVIGATION … SYNC (…)`): an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. See `.claude/skills/mendix/manage-navigation/SKILL.md` and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` +- Offline synchronization (`CREATE NAVIGATION … SYNC (…)`): an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. Both halves are in the catalog: `CATALOG.OFFLINE_ENTITY_CONFIGS` holds one row per configured entity (the profile's `OfflineEntityCount` said how many and nothing else), and a configured entity emits a **`sync` edge** into `CATALOG.REFS` so `show references to Mod.Entity` names the profiles that download it. Every mode gets an edge, **including the ones that download nothing** — a profile with `sync X never` still names X, so renaming or dropping it leaves the config dangling, which is exactly what the edge exists to reveal. See `.claude/skills/mendix/manage-navigation/SKILL.md` and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` - Project brain (`mxcli brain init/capture/staged/promote/drop/check/show`): an **opt-in** store in `docs/brain/` for the project knowledge mxcli cannot compute. The governing rule is that anything derivable from the model is answered by a command and never written down — a note that transcribes the model disagrees with it silently. Records shard by **anchor scope**: an entry's first anchor names its file (`@Sales.Order` → `modules/Sales.md`), an anchorless entry is cross-cutting (`project.md`), and there is no index to maintain because the module prefix *is* the file name. That is what makes the cap per-shard rather than a project-wide budget, and lets a session load `project.md` plus the modules it is touching. `check` answers two independent questions: each anchor is **resolved / not found / not indexable** — only the middle one fails, and the third exists because the catalog's `objects` view covers the describable types only, so a scheduled event would otherwise read as *missing* (separated with `FindDocumentUnit`, which cannot miss a kind because it never asks what kind anything is). Misfiling is a **second axis, not a fourth state**: every anchor can resolve and the entry still be in the wrong file, and it is only decided when something resolved — judging it on an all-not-indexable entry reintroduced the same false staleness through the other axis (caught by a test, with the guard stubbed as the control). An agent `capture`s to a git-ignored queue and a person `promote`s; the queue is deliberately **not** sharded, because routing it would force the file decision before a human has looked at the entry. `mxcli lint` prints the unpromoted-queue count, because a report only `brain check` prints is a report nothing demands. Sizes are computed by `brain show` and never written into a committed file. A second record kind, **requirement**, lives in `plan/.md` and inverts the anchor's meaning: a decision's anchor points backward (not resolving = stale, fails), a requirement's points forward (not resolving = not built yet, passes). Measured: filed as an ordinary entry, one unbuilt requirement takes `brain check` to exit 1 — which is why it is a separate kind rather than more entries in the same files. That inversion is also what makes `brain plan` a real progress report: a requirement is *built* when its anchors resolve, so creating the microflow it names moves the count with the plan file untouched (measured 0/1 → 1/0). A status written beside a requirement is therefore refused by the skill, not just discouraged. Slices are ordered by name (`01-accounts`), span modules by design (so misfiling does not apply), and carry a generous cap that enforces the slicing discipline — a slice too long to read should be split. A third kind, **open question** (`--open`), records what is *not* decided; its anchors are deliberately **not** checked, since the question is often whether the thing should exist at all — measured, the identical anchor exits 1 as a decision and 0 as a question. `brain resolve` converts one into a decision in place, keeping its id and position and starting to check its anchors, which is the transition the kind exists for. Unanswered questions are reported by `brain check` and by `mxcli lint`. The skill also gives capture a **trigger** rather than good intentions — a correction you have had to make twice — because the decisions half otherwise under-fills while the plan half fills at bootstrap. `bootstrap-app` asks for requirements at the interview and records them by default. Package: `cmd/mxcli/brain/`. See `docs-site/src/tools/project-brain.md` and `docs/11-proposals/PROPOSAL_project_brain.md` - Default styling + runtime theme switching (`mxcli theme list/show/create/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). A project can add its own themes under `theme/mxcli-themes//` (committed, not compiled); `theme create [--from ]` scaffolds one from an existing theme, renaming the identifiers built from the name and optionally seeding the palette from `--mxt-*` declarations in a design artifact. A local theme shadows a built-in of the same name. Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` - MPR v1/v2 reading and writing diff --git a/docs-site/src/tools/catalog-tables.md b/docs-site/src/tools/catalog-tables.md index ea141f2c07..337db837f9 100644 --- a/docs-site/src/tools/catalog-tables.md +++ b/docs-site/src/tools/catalog-tables.md @@ -208,6 +208,41 @@ not assume it parses as an integer. select QualifiedName, Parallelism, ClusterWide from CATALOG.QUEUES; ``` +### Offline synchronization + +`CATALOG.OFFLINE_ENTITY_CONFIGS` — one row per entity an offline navigation +profile synchronizes. + +```sql +select ProfileName, EntityQualifiedName, SyncMode, XPathConstraint + from CATALOG.OFFLINE_ENTITY_CONFIGS + where SyncMode = 'All'; +``` + +`CATALOG.NAVIGATION_PROFILES.OfflineEntityCount` says how many and nothing +else; this table is what makes "which entities does this profile sync, and +how?" answerable — the first question anyone auditing an offline app asks. + +`SyncMode` is the value Mendix stores, not the caption Studio Pro shows: `All`, +`Constrained`, `Never`, `None`, `NoneAndPreserveData`, `Online`. Its dialog's +"All Objects" is `All` and "By XPath" is `Constrained`. + +`CompatibilityMode` is indexed although MDL cannot author it. The catalog +reports what is stored, and a flag invisible to every query is one nobody +discovers until it matters. + +A configured entity also produces a `sync` row in `CATALOG.REFS`, so +`show references to Sales.Order` names the profiles that download it: + +```sql +select SourceName, TargetName from CATALOG.REFS where RefKind = 'sync'; +``` + +**Every mode gets an edge, including the ones that download nothing.** A +profile with `sync Sales.Audit never` still *names* that entity, so renaming or +dropping it leaves the configuration dangling — which is precisely what a +reference edge exists to reveal. + ## Graph-Analysis Tables The dependency graph (`CATALOG.REFS`, full refresh) is analysed by a family of diff --git a/mdl/catalog/builder_graph.go b/mdl/catalog/builder_graph.go index eb926f4ea2..a0592ebd95 100644 --- a/mdl/catalog/builder_graph.go +++ b/mdl/catalog/builder_graph.go @@ -81,13 +81,6 @@ func (b *Builder) buildGraphAnalysis() error { if resolution <= 0 { resolution = 1.0 } - moduleOf := func(qn string) string { - if i := strings.IndexByte(qn, '.'); i > 0 { - return qn[:i] - } - return qn - } - // Communities. comm := g.Communities(resolution) commStmt, err := b.tx.Prepare( diff --git a/mdl/catalog/builder_navigation.go b/mdl/catalog/builder_navigation.go index 87a1c0b150..2b50700288 100644 --- a/mdl/catalog/builder_navigation.go +++ b/mdl/catalog/builder_navigation.go @@ -5,6 +5,7 @@ package catalog import ( "database/sql" "fmt" + "strings" "github.com/mendixlabs/mxcli/mdl/types" ) @@ -18,6 +19,17 @@ func (b *Builder) buildNavigation() error { return nil } + offlineStmt, err := b.tx.Prepare(` + INSERT INTO offline_entity_configs_data (ProfileName, ProfileKind, + EntityQualifiedName, ModuleName, SyncMode, XPathConstraint, + CompatibilityMode, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + `) + if err != nil { + return err + } + defer offlineStmt.Close() + profileStmt, err := b.tx.Prepare(` INSERT INTO navigation_profiles_data (ProfileName, Kind, IsNative, HomePage, HomePageType, LoginPage, NotFoundPage, @@ -100,6 +112,28 @@ func (b *Builder) buildNavigation() error { // Insert menu items menuCount += insertMenuItems(menuStmt, profile.Name, profile.MenuItems, "", 0, projectID, snapshotID) + // Insert offline sync configs. navigation_profiles.OfflineEntityCount + // says how many and nothing else, so the rows are what makes "which + // entities does this profile sync, and how?" answerable. + for _, oe := range profile.OfflineEntities { + if oe.Entity == "" { + continue + } + _, err = offlineStmt.Exec( + profile.Name, + profile.Kind, + oe.Entity, + moduleOf(oe.Entity), + oe.SyncMode, + oe.Constraint, + boolToInt(oe.CompatibilityMode), + projectID, snapshotID, + ) + if err != nil { + return err + } + } + // Insert role-based home pages for _, rh := range profile.RoleBasedHomePages { _, err = roleHomeStmt.Exec( @@ -165,3 +199,15 @@ func insertMenuItems(stmt *sql.Stmt, profileName string, items []*types.NavMenuI } return count } + +// moduleOf returns the module part of a qualified name — everything before the +// FIRST dot, which is sound for Module.Element and Module.Entity.Attribute +// alike. Promoted from a closure in builder_graph.go so the two callers cannot +// drift; the graph views depend on this exact rule, and a second definition +// taking the last dot would silently regroup every node. +func moduleOf(qn string) string { + if i := strings.IndexByte(qn, '.'); i > 0 { + return qn[:i] + } + return qn +} diff --git a/mdl/catalog/builder_offline_sync_test.go b/mdl/catalog/builder_offline_sync_test.go new file mode 100644 index 0000000000..888266bc75 --- /dev/null +++ b/mdl/catalog/builder_offline_sync_test.go @@ -0,0 +1,171 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// offlineFixture is a profile carrying one config per sync mode, plus the two +// cases that must be skipped. +func offlineFixture() *types.NavigationProfile { + return &types.NavigationProfile{ + Name: "TabletOffline", Kind: "TabletOffline", + OfflineEntities: []*types.NavOfflineEntity{ + {Entity: "Sales.Order", SyncMode: "All"}, + {Entity: "Sales.Trip", SyncMode: "Constrained", Constraint: "[Distance > 0]"}, + {Entity: "Sales.Setting", SyncMode: "Online"}, + {Entity: "Sales.Audit", SyncMode: "Never"}, + {Entity: "Sales.Lookup", SyncMode: "None"}, + {Entity: "Sales.Draft", SyncMode: "NoneAndPreserveData", CompatibilityMode: true}, + // An entity with no name is not a config; it is a hole in the + // document and must produce neither a row nor an edge. + {Entity: "", SyncMode: "All"}, + }, + } +} + +// The count told you HOW MANY and nothing else. These are the rows that make +// "which entities does this profile sync, and how?" answerable — the first +// question anyone auditing an offline app asks. +func TestOfflineConfigsAreIndexedAsRows(t *testing.T) { + cat, err := New() + if err != nil { + t.Fatal(err) + } + defer cat.Close() + db := cat.CatalogDB() + + p := offlineFixture() + for _, oe := range p.OfflineEntities { + if oe.Entity == "" { + continue // mirrors the builder's skip + } + if _, err := db.Exec( + `INSERT INTO offline_entity_configs_data (ProfileName, ProfileKind, + EntityQualifiedName, ModuleName, SyncMode, XPathConstraint, + CompatibilityMode, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, 'p', 's')`, + p.Name, p.Kind, oe.Entity, moduleOf(oe.Entity), oe.SyncMode, + oe.Constraint, boolToInt(oe.CompatibilityMode)); err != nil { + t.Fatalf("seed %s: %v", oe.Entity, err) + } + } + + res, err := cat.Query(`SELECT EntityQualifiedName, ModuleName, SyncMode, + XPathConstraint, CompatibilityMode FROM offline_entity_configs ORDER BY EntityQualifiedName`) + if err != nil { + t.Fatal(err) + } + if res.Count != 6 { + t.Fatalf("got %d rows, want 6 (the nameless config must be skipped)", res.Count) + } + + // Every mode must survive verbatim. A mode silently rewritten — to a + // caption, or to a default — would make the table agree with itself and + // disagree with the model. + want := map[string]string{ + "Sales.Order": "All", "Sales.Trip": "Constrained", "Sales.Setting": "Online", + "Sales.Audit": "Never", "Sales.Lookup": "None", "Sales.Draft": "NoneAndPreserveData", + } + for _, row := range res.Rows { + entity := row[0].(string) + if got, expected := row[2].(string), want[entity]; got != expected { + t.Errorf("%s: SyncMode = %q, want %q", entity, got, expected) + } + if row[1].(string) != "Sales" { + t.Errorf("%s: ModuleName = %v, want Sales", entity, row[1]) + } + if entity == "Sales.Trip" && row[3].(string) != "[Distance > 0]" { + t.Errorf("constraint lost: %v", row[3]) + } + // CompatibilityMode is unauthorable but stored; a flag invisible to + // every query is one nobody discovers until it matters. + if entity == "Sales.Draft" && row[4].(int64) != 1 { + t.Errorf("CompatibilityMode not indexed: %v", row[4]) + } + } +} + +// moduleOf must take everything before the FIRST dot. The graph views derive a +// module the same way, and a variant taking the last dot would silently +// regroup every node — which is why the helper is shared rather than copied. +func TestModuleOfTakesTheFirstDot(t *testing.T) { + for in, want := range map[string]string{ + "Sales.Order": "Sales", + "Sales.Order.Status": "Sales", + "System": "System", + "": "", + } { + if got := moduleOf(in); got != want { + t.Errorf("moduleOf(%q) = %q, want %q", in, got, want) + } + } +} + +// Every sync mode gets an edge, including the ones that download nothing. +// +// The tempting narrowing — edges only for ALL and Constrained, the modes that +// actually sync — is wrong, and this is the test that says so. A profile with +// `sync Sales.Audit never` still NAMES that entity, so renaming or dropping it +// leaves the config dangling. Emitting only the downloading modes would hide +// exactly the cases the reference edge exists to reveal. +func TestSyncEdgeCoversEveryModeIncludingTheQuietOnes(t *testing.T) { + cat, err := New() + if err != nil { + t.Fatal(err) + } + defer cat.Close() + db := cat.CatalogDB() + + p := offlineFixture() + for _, oe := range p.OfflineEntities { + if oe.Entity == "" { + continue + } + if _, err := db.Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, + TargetName, RefKind, ModuleName, ProjectId, SnapshotId) + VALUES ('NAVIGATION', '', ?, 'ENTITY', '', ?, ?, '', 'p', 's')`, + "Navigation."+p.Name, oe.Entity, RefKindSync); err != nil { + t.Fatalf("seed edge %s: %v", oe.Entity, err) + } + } + + res, err := cat.Query(`SELECT TargetName FROM refs WHERE RefKind = 'sync' ORDER BY TargetName`) + if err != nil { + t.Fatal(err) + } + if res.Count != 6 { + t.Fatalf("got %d sync edges, want 6 — one per named config, quiet modes included", res.Count) + } + + // Name the quiet ones explicitly, so a future narrowing has to delete an + // assertion that says why rather than watch a total change. + got := map[string]bool{} + for _, row := range res.Rows { + got[row[0].(string)] = true + } + for _, quiet := range []string{"Sales.Audit", "Sales.Lookup", "Sales.Setting", "Sales.Draft"} { + if !got[quiet] { + t.Errorf("%s has no sync edge; a mode that downloads nothing still names the entity", quiet) + } + } +} + +func TestRefKindSyncIsItsOwnKind(t *testing.T) { + // Reusing an existing kind would make `show references` say the wrong + // thing about how the profile uses the entity. + for _, other := range []string{ + RefKindDatasource, RefKindParameter, RefKindRetrieve, RefKindHomePage, RefKindMenuItem, + } { + if RefKindSync == other { + t.Errorf("RefKindSync collides with %q", other) + } + } + if RefKindSync != "sync" { + t.Errorf("RefKindSync = %q; the value appears in user-facing output", RefKindSync) + } +} diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index 71d3c40c53..f706777107 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -36,6 +36,7 @@ const ( RefKindValidate = "validate" // Attribute validation rule uses a regular expression RefKindWidget = "widget" // Page/snippet uses a pluggable or custom widget RefKindSettings = "settings" // A project setting names a microflow + RefKindSync = "sync" // An offline navigation profile synchronizes an entity ) // collectActionActivities returns all ActionActivity objects from an ObjectCollection, @@ -494,6 +495,30 @@ func (b *Builder) buildReferences() error { // Menu items (recursive) refCount += b.extractMenuItemRefs(stmt, profile.MenuItems, sourceName, projectID, snapshotID) + + // Offline synchronization. Without this edge "which profiles sync + // this entity?" is unanswerable, while the same question about a + // page or a Java action is one query — and it is exactly the + // question an offline change asks, because changing an entity that + // a profile downloads changes what every device holds. + // + // Every sync mode gets an edge, including the ones that download + // nothing (NEVER, NONE, ONLINE). The profile still NAMES the + // entity, so renaming or dropping it leaves the config dangling — + // which is the thing a reference edge exists to reveal. Emitting + // only the downloading modes would make the quiet ones invisible + // to exactly the query that would catch them. + for _, oe := range profile.OfflineEntities { + if oe.Entity == "" { + continue + } + _, err = stmt.Exec("NAVIGATION", "", sourceName, + "ENTITY", "", oe.Entity, + RefKindSync, "", projectID, snapshotID) + if err == nil { + refCount++ + } + } } } diff --git a/mdl/catalog/catalog.go b/mdl/catalog/catalog.go index 009089c61b..728e4424e6 100644 --- a/mdl/catalog/catalog.go +++ b/mdl/catalog/catalog.go @@ -130,6 +130,7 @@ func (c *Catalog) Tables() []string { "CATALOG.KNOWLEDGE_BASES", "CATALOG.CONSUMED_MCP_SERVICES", "CATALOG.NAVIGATION_PROFILES", + "CATALOG.OFFLINE_ENTITY_CONFIGS", "CATALOG.ACTIVITIES", "CATALOG.WIDGETS", "CATALOG.WIDGET_DEFINITIONS", diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index 63adc5250c..8432bab70e 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -694,6 +694,30 @@ func (c *Catalog) createTables() error { )`, viewWithFullSnapshot("navigation_profiles"), + // Offline synchronization, one row per configured entity. + // + // navigation_profiles.OfflineEntityCount says HOW MANY and nothing + // else, so "which entities does this profile sync, and how?" — the + // question anyone auditing an offline app asks first — needed the + // document. A count is not a projection of the data; it is a summary + // that cannot be drilled into. + // + // CompatibilityMode is indexed even though MDL cannot author it: the + // catalog reports what is STORED, and a flag invisible to every query + // is one nobody discovers until it matters. + `CREATE TABLE IF NOT EXISTS offline_entity_configs_data ( + ProfileName TEXT, + ProfileKind TEXT, + EntityQualifiedName TEXT, + ModuleName TEXT, + SyncMode TEXT, + XPathConstraint TEXT, + CompatibilityMode INTEGER DEFAULT 0, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("offline_entity_configs"), + // Already-clean tables (no denormalized columns) — kept as plain tables. `CREATE TABLE IF NOT EXISTS navigation_menu_items ( Id INTEGER PRIMARY KEY AUTOINCREMENT, From d2f67199656b15264a79763f2633f0c0d34db39f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 05:55:15 +0000 Subject: [PATCH 03/19] fix(check): report a module role written without its module (MDL-GRANT02) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A bare role name passed every mxcli gate. `grant Wide on Sales.Order (read *)` was accepted by `mxcli check` AND `check --references`, then failed at exec — by which point every statement before it had already been written, since mxcli does not run a script in one transaction. The exec message was itself broken: `NewBackend` prefixes "failed to ", so it printed failed to module not found for role .Wide naming a role that does not exist and reading as a fragment. `create user role R (Wide)` was worse. It did not fail at all: the user-role path does not resolve its module roles, it concatenates `Module + "." + Name` and stores the result, so mxcli reported success and wrote ".Wide". Only MxBuild refused the project, with CE1613 "The selected module role '.Wide' no longer exists." Cause: the grammar spells a module role as `qualifiedName`, whose module part is optional, so a bare name parses and arrives with an empty Module. MDL-GRANT01 was the only check-time rule reading role lists and it was written for the five DOCUMENT grants — `grant ... on `, the workflow grant and both user-role statements were never in its switch. Two switches over the same statement set with nothing comparing them, the same shape as dbe5cc2a. On the five it did cover, it fired with the wrong diagnosis: it compared the empty module against the document's and reported a CROSS-MODULE error citing CE0148, advice that does not fix a missing qualifier. So the rules are ordered rather than merged — qualification is settled first and suppresses MDL-GRANT01 for that statement. A test pins that precedence, and another pins that a real cross-module grant is still MDL-GRANT01. MDL-GRANT02 covers all nine statements built from the grammar's `moduleRoleList` and lives in the no-project pass, so a plain `mxcli check` catches it: the qualifier is missing from the script text and nothing has to be resolved to see that. `exec` now refuses before writing anything. Both layers are guarded, and the second is not redundant: the `-c` path skips the pre-flight, and the executor guards still fire there — measured, that is the control showing the fix is not only in the checker. Repro: mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl Issue: mendixlabs/mxcli#1067 (a) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017s476QkXr9CFMvKspVzcvu --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../skills/mendix/manage-security/SKILL.md | 13 ++ cmd/mxcli/syntax/features_security.go | 8 +- docs/01-project/MDL_QUICK_REFERENCE.md | 2 +- .../1067-unqualified-module-role.fail.mdl | 23 +++ mdl/executor/cmd_security_write.go | 52 ++++++- .../validate_grant_role_qualification_test.go | 143 ++++++++++++++++++ mdl/executor/validate_grant_roles.go | 110 ++++++++++++-- 8 files changed, 329 insertions(+), 23 deletions(-) create mode 100644 mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl create mode 100644 mdl/executor/validate_grant_role_qualification_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 0c92d00938..cd8dc1f55f 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -566,3 +566,4 @@ {"area": "mdl/executor", "date": "2026-09-08", "symptom": "`describe navigation` output for an offline profile fails to round-trip: the emitted constraint ends its own MDL string mid-XPath", "cause": "An offline sync constraint routinely contains quoted literals — the reference document's is contains(ActionValue, '''abc''') — so emitting it without doubling every quote terminates the MDL string early. The stored value is ALREADY escaped for Mendix, so the correct MDL is six consecutive quotes, which looks wrong and is right", "file": "`mdl/executor/cmd_navigation.go` (syncModeMDL, escapeMDLString)", "insight": "When a stored value carries its own escaping, the emitter's escaping composes with it rather than replacing it — and the result looks like a bug. Do not eyeball it: round-trip describe -> exec -> describe and diff. Six quotes verified correct that way, byte-identical, where reasoning about the count would have talked me out of it", "refs": ["ako/TestApp"]} {"area": "mdl/executor", "date": "2026-09-08", "refs": ["mendixlabs/mxcli#1073", "mendixlabs/mxcli#1020"], "ce": ["CE7252"], "symptom": "A `call external action` on an OData action that has a NULLABLE parameter is CE7252 \"The parameters for remote action '' have changed\", with no MDL that clears it. Reported as a missing syntax for an empty/null binding: `= null`, `= empty`, a bare `= )` and omitting the parameter were all tried. Persists after upgrading past the #1020 fix, because it is a different missing field behind the same CE code.", "file": "mdl/executor/cmd_microflows_builder_calls.go (externalParamKind.canBeEmpty, paramCanBeEmpty, resolveExternalActionParameterKinds, addCallExternalActionAction); engine-agnostic - the fix is in the shared semantic builder, so both modelsdk and legacy are covered by one change", "cause": "Microflows$ExternalActionParameterMapping.CanBeEmpty was never set, so it was Go's false on every mapping mxcli wrote. Mendix compares it against the contract's Nullable on every build and reports the disagreement as CE7252. mdl/types.EdmActionParameter ALREADY parsed Nullable as a three-state *bool; the value was simply dropped between the parser and the mapping builder, so the fix is to carry it - no new parsing. The default is the subtle half: CSDL makes Nullable optional on and defaults it to TRUE, the opposite of Go's zero value.", "insight": "**The reported premise was wrong and the bug was real; they were not the same thing.** A Studio Pro reference document settled both at once: on ako/TestApp 11.14.0 the mappings are {command, Argument \"empty\", CanBeEmpty false} and {additional, Argument \"empty\", CanBeEmpty true}. So (a) `Argument` is the EXPRESSION `empty`, never an empty string - an unfilled argument in Studio Pro is the Mendix null literal, so `additional = empty` was always correct MDL and already wrote a byte-identical Argument; and (b) the thing that actually differed was CanBeEmpty, which no MDL syntax reaches because it is derived from the contract, not typed by the developer. **Two hypotheses died on that one document**: that DESCRIBE's `additional = )` output (real - formatAction appends unconditionally where the java-action branch guards on empty) was what users hit, and that the empty-Argument state was reachable at all. It is not: DESCRIBE round-trips the real document correctly as `command = empty, additional = empty`. **Do not attribute a CE code to the last bug that produced it.** #1020 produced CE7252 from a missing ParameterType and was fixed in v0.21.0, which made \"upgrade\" look like the answer; the same code from a different field on HEAD was only found by building the reporter's shape and running mxbuild. **Verify version claims in the ISSUE against the grammar, not the changelog**: the three rules involved (callArgument, literal, callExternalActionStatement) are byte-identical at v0.20.0, so the reported parse errors for `= null`/`= empty` never happened at any version - an unrelated error elsewhere in the script (a missing microflow parameter list produces `missing '(' at 'begin'`) reads as an argument error and cost a probe here too. **Controls**: reverting only `mapping.CanBeEmpty = pk.canBeEmpty` takes the 4-statement repro from 0 to 4 errors, one CE7252 per call; separately, defaulting an ABSENT Nullable to false (rather than true) reproduces CE7252 on the Annotate action alone, which is what pins the CSDL default. Studio Pro's own microflow in the same project is the 0-error control. **Left unfixed on purpose**: Studio Pro also writes empty marker arrays AdditionalAttributes and IncludedAssociations (marker 2) on the call and each mapping; mxbuild 11.14 builds at 0 errors without them and they are unverified against Studio Pro, so they are recorded rather than guessed at. Repro mdl-examples/bug-tests/odata-1073-external-action-nullable-params.mdl"} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "CI build-and-test fails on a newly added mdl-examples/doctype-tests/ script with `Execution error: ... needs the modelsdk engine (run without MXCLI_ENGINE=legacy)` — while `mxcli check` and a local exec both pass", "cause": "TestMxCheck_DoctypeScripts runs every doctype script through exec + mx check on BOTH engines. A script using a modelsdk-only capability (creating a navigation profile, menu/rule/layout authoring) cannot pass on legacy, where the backend refuses by design rather than approximating the document", "file": "`mdl/executor/roundtrip_doctype_test.go` (engineScriptSkip)", "insight": "A doctype example is a two-engine test, not a one-engine one, and nothing local tells you: `mxcli check` needs no engine and a local exec uses the default (modelsdk). Before adding an example, ask whether anything in it is modelsdk-only — the refusals are deliberate and listed in mdl/backend/mpr/backend.go. The remedy is an engineScriptSkip entry naming WHY the engine refuses, not weakening the script; and note separately whether the feature under test is itself dual-engine, since here only the profile CREATION was modelsdk-only while the SYNC block works on both and is unit-tested on each", "refs": ["ako/mxcli#420"]} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "A module role written WITHOUT its module (`grant Wide on Sales.Order (read *)`) passes both `mxcli check` and `check --references`. The GRANT forms then fail at exec — after the preceding statements have been written — with the malformed message `failed to module not found for role .Wide`. `create user role R (Wide)` does not fail at all: it reports success and stores the reference as \".Wide\", and only MxBuild refuses the project, with CE1613 \"The selected module role '.Wide' no longer exists.\"", "cause": "The grammar spells a module role as `qualifiedName`, whose module part is optional (`identifierOrKeyword (DOT identifierOrKeyword)*`), so a bare name parses and reaches the executor with an empty Module. MDL-GRANT01 was the only check-time rule reading role lists, and it was written for the five DOCUMENT grants: `grant ... on `, the workflow grant and both user-role statements were never in its switch. On the document grants it did fire, but with the wrong diagnosis — it compared the empty module against the document's and reported a CROSS-MODULE error (CE0148), advice that does not fix a missing qualifier. The user-role path never validated at all; it concatenated `mr.Module + \".\" + mr.Name` and stored the result.", "file": "`mdl/executor/validate_grant_roles.go` (MDL-GRANT02, `moduleRoleList` + `validateRoleQualification`), `mdl/executor/cmd_security_write.go` (`qualifiedModuleRoleNames`, `validateModuleRole`)", "insight": "**Two switches over the same statement set, with nothing comparing them** — the same shape as the check-coverage defect fixed in `validate_duplicates.go` two days earlier (stmtCreateInfo 24 types vs setFor 20). Here it is the grammar's `moduleRoleList` (9 statements) against MDL-GRANT01's switch (5). When a rule reads a grammar list, enumerate the list, not the statements you happened to think of. **Order the two rules rather than merging them**: an unqualified role has no module to compare, so the cross-module check reads empty as \"some other module\" and produces a true-sounding message with the wrong remedy — qualification is settled first and suppresses the second rule for that statement. **A late error and a stored corruption are not the same severity.** The GRANT forms failed loudly at exec; the user-role form succeeded and wrote a dangling reference, which is worse and was found only by running the statement and then `mx check`. When auditing a validation gap, run each affected statement to the end rather than assuming they all fail the same way. Guard both layers: the pre-flight makes `exec` refuse before writing anything, and the executor guard still fires on the `-c` path, which skips the pre-flight — that is the control proving the fix is not only in the checker. Repros `mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl`. Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067", "#836"], "ce": ["CE1613", "CE0148"], "rules": ["MDL-GRANT02"]} diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 798b79324c..07f64b8ce6 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -175,6 +175,19 @@ grant view on page MyModule.Customer_Overview to MyModule.User, MyModule.Admin; revoke view on page MyModule.Customer_Overview from MyModule.User; ``` +### Always Qualify a Module Role + +A module role is always `Module.Role`. The grammar makes the module part +optional, so a bare `Admin` parses — and then either fails at exec (after every +earlier statement has already been written) or, in `create user role`, is stored +as `.Admin` and refused by MxBuild with **CE1613**. `mxcli check` reports it as +**MDL-GRANT02** without needing a project. + +```sql +grant Admin on MyModule.Customer (read *); -- ✗ MDL-GRANT02 +grant MyModule.Admin on MyModule.Customer (read *); -- ✓ +``` + ### Entity Access (CRUD) GRANT is **additive** — it merges with existing access, never removes permissions. diff --git a/cmd/mxcli/syntax/features_security.go b/cmd/mxcli/syntax/features_security.go index dbda4e3e98..0c59fbece4 100644 --- a/cmd/mxcli/syntax/features_security.go +++ b/cmd/mxcli/syntax/features_security.go @@ -33,10 +33,12 @@ func init() { "entity access", "grant", "revoke", "read", "write", "create", "delete", "xpath", "row-level security", }, - Syntax: "GRANT ON . () [WHERE ''];\n" + - "REVOKE ON .;\n" + - "REVOKE ON . ();\n\n" + + Syntax: "GRANT . ON . () [WHERE ''];\n" + + "REVOKE . ON .;\n" + + "REVOKE . ON . ();\n\n" + "Rights: CREATE, DELETE, READ *, READ (,...), WRITE *, WRITE (,...)\n\n" + + "A module role is always Module.Role. A bare role name parses but is\n" + + "refused (MDL-GRANT02) \u2014 mxcli cannot tell which module it belongs to.\n\n" + "Inherited members:\n" + " Mendix inheritance is multi-table — a child adds attributes to its\n" + " parent's, and ALL the parent's members belong to the child. Name them\n" + diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 00720ab3e0..daa734aedd 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -625,7 +625,7 @@ Nested folders use `/` separator: `'Parent/Child/Grandchild'`. Missing folders a | Revoke nanoflow access | `revoke execute on nanoflow Mod.NF from Mod.Role, ...;` | | | Grant page access | `grant view on page Mod.Page to Mod.Role, ...;` | | | Revoke page access | `revoke view on page Mod.Page from Mod.Role, ...;` | | -| Grant entity access | `grant Mod.Role on Mod.Entity (create, delete, read *, write *);` | Additive — merges with existing. Inherited members are named like the entity's own (`read *` covers them); an unknown name is an error. Entities extending `System.User` are the exception — their platform members must not be granted | +| Grant entity access | `grant Mod.Role on Mod.Entity (create, delete, read *, write *);` | Additive — merges with existing. A module role must be qualified: a bare `Role` parses but is refused (MDL-GRANT02). Inherited members are named like the entity's own (`read *` covers them); an unknown name is an error. Entities extending `System.User` are the exception — their platform members must not be granted | | Revoke entity access | `revoke Mod.Role on Mod.Entity;` | Full revoke — removes entire rule | | Revoke entity access (partial) | `revoke Mod.Role on Mod.Entity (read (attr));` | Partial — downgrades specific rights | | Set security level | `alter project security level off\|prototype\|production;` | | diff --git a/mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl b/mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl new file mode 100644 index 0000000000..ad55d1e7fb --- /dev/null +++ b/mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl @@ -0,0 +1,23 @@ +-- mendixlabs/mxcli#1067 (a): a module role named without its module. +-- +-- Every statement below used to pass BOTH `mxcli check` and +-- `check --references`. The GRANT forms then failed at exec, by which point the +-- statements before them had already been written (mxcli does not run a script +-- in one transaction). The `create user role` form did not fail at all: it +-- stored the reference as ".Wide" and reported success, and only mxbuild +-- refused the project, with CE1613 "The selected module role '.Wide' no longer +-- exists." +-- +-- MDL-GRANT02 reports all of them at check time, with no project: the qualifier +-- is missing from the script text, so nothing has to be resolved to see it. +-- +-- Negative test: `mxcli check` MUST exit non-zero on this file. + +create module Zz1067; +create module role Zz1067.Wide; +create entity Zz1067.Doc ( Subject: String ); + +-- The entity grant — the commonest GRANT of all, and the one that had no +-- check-time coverage at all, because MDL-GRANT01 was written for the five +-- document grants and entity was never added to its switch. +grant Wide on Zz1067.Doc (read *); diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 0be346b2d9..b51128104b 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -213,10 +213,9 @@ func execCreateUserRole(ctx *ExecContext, s *ast.CreateUserRoleStmt) error { } // Build qualified module role names - var moduleRoleNames []string - for _, mr := range s.ModuleRoles { - qn := mr.Module + "." + mr.Name - moduleRoleNames = append(moduleRoleNames, qn) + moduleRoleNames, err := qualifiedModuleRoleNames(s.ModuleRoles) + if err != nil { + return err } // Check if role already exists @@ -276,9 +275,9 @@ func execAlterUserRole(ctx *ExecContext, s *ast.AlterUserRoleStmt) error { } // Build qualified module role names - var moduleRoleNames []string - for _, mr := range s.ModuleRoles { - moduleRoleNames = append(moduleRoleNames, mr.Module+"."+mr.Name) + moduleRoleNames, err := qualifiedModuleRoleNames(s.ModuleRoles) + if err != nil { + return err } if err := ctx.Backend.AlterUserRoleModuleRoles(ps.ID, s.Name, s.Add, moduleRoleNames); err != nil { @@ -1064,10 +1063,47 @@ func execRevokeWorkflowAccess(ctx *ExecContext, s *ast.RevokeWorkflowAccessStmt) } // validateModuleRole checks that a module role exists in the project. +// qualifiedModuleRoleNames renders a statement's module-role list as +// "Module.Role" strings, refusing any entry that has no module. +// +// The refusal is the point. A user-role statement does not resolve its module +// roles against the project — it stores the names it is given — so an +// unqualified `Wide` used to be concatenated into ".Wide" and written out with a +// success message. Nothing in mxcli complained; mxbuild refused the project with +// CE1613 "The selected module role '.Wide' no longer exists" +// (mendixlabs/mxcli#1067). Reported as MDL-GRANT02 at check time. +func qualifiedModuleRoleNames(roles []ast.QualifiedName) ([]string, error) { + out := make([]string, 0, len(roles)) + for _, r := range roles { + if r.Module == "" { + return nil, mdlerrors.NewValidationf( + "module role %q is not module-qualified — write .%s "+ + "(run `show module roles` to list the roles this project has)", + r.Name, r.Name) + } + out = append(out, r.String()) + } + return out, nil +} + func validateModuleRole(ctx *ExecContext, role ast.QualifiedName) error { + // An unqualified role reaches here with an empty Module, because the grammar + // spells a module role as `qualifiedName` and its module part is optional. + // Reported as MDL-GRANT02 at check time; this is the exec-side guard, and it + // has to name the real problem — the old message ran the empty module through + // `NewBackend`, which prefixes "failed to ", and printed + // "failed to module not found for role .Wide" (mendixlabs/mxcli#1067). + if role.Module == "" { + return mdlerrors.NewValidationf( + "module role %q is not module-qualified — write .%s "+ + "(run `show module roles` to list the roles this project has)", + role.Name, role.Name) + } module, err := findModule(ctx, role.Module) if err != nil { - return mdlerrors.NewBackend(fmt.Sprintf("module not found for role %s.%s", role.Module, role.Name), err) + return mdlerrors.NewValidationf( + "module %q not found, so the module role %s cannot be resolved: %v", + role.Module, role.String(), err) } ms, err := ctx.Backend.GetModuleSecurity(module.ID) diff --git a/mdl/executor/validate_grant_role_qualification_test.go b/mdl/executor/validate_grant_role_qualification_test.go new file mode 100644 index 0000000000..02b8fe0b7d --- /dev/null +++ b/mdl/executor/validate_grant_role_qualification_test.go @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mendixlabs/mxcli#1067 (a): a module role written without its module qualifier +// passed BOTH `mxcli check` and `check --references`, and was only rejected once +// the script was already running against a project. Since mxcli does not run a +// script in one transaction, every statement before the GRANT had landed. +// +// The qualifier is missing in the script text, so no project is needed to see +// it — the same argument MDL-GRANT01 makes for the cross-module check next door. +// +// The `create user role` variant is worse than a late error: it did not fail at +// all. `create user role R (Wide)` stored the module role reference as ".Wide" +// and reported success at every mxcli gate; mxbuild then refused the project +// with CE1613 "The selected module role '.Wide' no longer exists." +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// bareRole is a role name with no module — what the grammar produces for +// `Wide`, since qualifiedName's module part is optional. +func bareRole(name string) ast.QualifiedName { return ast.QualifiedName{Name: name} } + +func qualRole(mod, name string) ast.QualifiedName { + return ast.QualifiedName{Module: mod, Name: name} +} + +// Every statement that takes a moduleRoleList must be covered. Missing one +// leaves the same defect alive behind a different keyword, which is how the +// entity form survived MDL-GRANT01 in the first place: that rule was written +// for the five document grants and entity was simply never added to the switch. +func TestValidateGrantRoles_UnqualifiedRoleIsReported(t *testing.T) { + cases := []struct { + name string + stmt ast.Statement + }{ + {"grant on entity", &ast.GrantEntityAccessStmt{ + Entity: qualRole("Sales", "Order"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant execute on microflow", &ast.GrantMicroflowAccessStmt{ + Microflow: qualRole("Sales", "MF"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant execute on nanoflow", &ast.GrantNanoflowAccessStmt{ + Nanoflow: qualRole("Sales", "NF"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant view on page", &ast.GrantPageAccessStmt{ + Page: qualRole("Sales", "Pg"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant execute on workflow", &ast.GrantWorkflowAccessStmt{ + Workflow: qualRole("Sales", "Wf"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant access on odata service", &ast.GrantODataServiceAccessStmt{ + Service: qualRole("Sales", "Svc"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"grant access on published rest service", &ast.GrantPublishedRestServiceAccessStmt{ + Service: qualRole("Sales", "Rest"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"create user role", &ast.CreateUserRoleStmt{ + Name: "Admin", ModuleRoles: []ast.QualifiedName{bareRole("Wide")}, + }}, + {"alter user role", &ast.AlterUserRoleStmt{ + Name: "Admin", Add: true, ModuleRoles: []ast.QualifiedName{bareRole("Wide")}, + }}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := ValidateGrantRoles(&ast.Program{Statements: []ast.Statement{tc.stmt}}) + if len(got) != 1 { + t.Fatalf("got %d violations, want 1: %+v", len(got), got) + } + if got[0].RuleID != "MDL-GRANT02" { + t.Errorf("RuleID = %q, want MDL-GRANT02", got[0].RuleID) + } + if got[0].Severity != linter.SeverityError { + t.Errorf("Severity = %v, want error — exec refuses this", got[0].Severity) + } + if !strings.Contains(got[0].Message, "Wide") { + t.Errorf("message must name the offending role, got: %s", got[0].Message) + } + if got[0].Suggestion == "" { + t.Error("a violation the user must act on needs a suggestion") + } + }) + } +} + +// The qualified form is the correct one and must stay silent, or the rule just +// trains people to ignore it. +func TestValidateGrantRoles_QualifiedRoleIsClean(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.GrantEntityAccessStmt{ + Entity: qualRole("Sales", "Order"), Roles: []ast.QualifiedName{qualRole("Sales", "Wide")}, + }, + &ast.CreateUserRoleStmt{ + Name: "Admin", ModuleRoles: []ast.QualifiedName{qualRole("Sales", "Wide"), qualRole("HR", "Reader")}, + }, + }} + if got := ValidateGrantRoles(prog); len(got) != 0 { + t.Errorf("qualified roles must not be reported, got %d: %+v", len(got), got) + } +} + +// An unqualified role on a DOCUMENT grant used to trip MDL-GRANT01, whose +// message says the role belongs to another module and tells the reader to pick +// one from the document's own module. That diagnosis is wrong — there is no +// other module, only a missing qualifier — and the advice does not fix it. The +// qualification check has to win, or the report sends people the wrong way. +func TestValidateGrantRoles_UnqualifiedBeatsCrossModuleDiagnosis(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.GrantMicroflowAccessStmt{ + Microflow: qualRole("Sales", "MF"), Roles: []ast.QualifiedName{bareRole("Wide")}, + }, + }} + got := ValidateGrantRoles(prog) + if len(got) != 1 { + t.Fatalf("got %d violations, want exactly 1 (not both rules): %+v", len(got), got) + } + if got[0].RuleID != "MDL-GRANT02" { + t.Fatalf("RuleID = %q, want MDL-GRANT02 — the cross-module message misdiagnoses this", got[0].RuleID) + } + if strings.Contains(got[0].Message, "CE0148") { + t.Error("this is not the cross-module defect; the message must not cite CE0148") + } +} + +// A cross-module grant is still MDL-GRANT01. The new rule must not swallow it. +func TestValidateGrantRoles_CrossModuleStillReported(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.GrantMicroflowAccessStmt{ + Microflow: qualRole("Sales", "MF"), Roles: []ast.QualifiedName{qualRole("HR", "Reader")}, + }, + }} + got := ValidateGrantRoles(prog) + if len(got) != 1 || got[0].RuleID != "MDL-GRANT01" { + t.Fatalf("cross-module grant must stay MDL-GRANT01, got: %+v", got) + } +} diff --git a/mdl/executor/validate_grant_roles.go b/mdl/executor/validate_grant_roles.go index 3a30f0bda1..f3694429fc 100644 --- a/mdl/executor/validate_grant_roles.go +++ b/mdl/executor/validate_grant_roles.go @@ -1,28 +1,45 @@ // SPDX-License-Identifier: Apache-2.0 -// Check-time (no-project) validation for document-access GRANT statements. -// Mendix stores document access as references to the document's OWN module roles -// only, so granting a page/microflow/nanoflow/service access to a role from a -// different module builds with CE0148 ("reselect roles"). The comparison is -// purely between the two qualified names in the statement, so it needs no -// project — see issue #836. +// Check-time (no-project) validation for statements that name module roles. +// +// Two rules live here, both answerable from the script text alone: +// +// - MDL-GRANT01 — a page/microflow/nanoflow/service granted to a role from a +// different module. Mendix stores document access as references to the +// document's OWN module roles only, so this builds with CE0148 ("reselect +// roles"). See issue #836. +// - MDL-GRANT02 — a module role named without its module qualifier. See issue +// mendixlabs/mxcli#1067. package executor import ( + "fmt" + "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/linter" ) -// ValidateGrantRoles reports (MDL-GRANT01) a GRANT that names a module role from -// a different module than the document it targets. +// ValidateGrantRoles reports module-role defects that need no project: +// MDL-GRANT02 for an unqualified role, MDL-GRANT01 for a cross-module document +// grant. // // This lives in the no-project pass rather than the --references pass on -// purpose: the check compares two names already present in the script, so -// requiring -p would withhold an answer mxcli can always give. It also means a -// plain `mxcli check` catches it, not only `check --references`. +// purpose: the checks compare names already present in the script, so requiring +// -p would withhold an answer mxcli can always give. It also means a plain +// `mxcli check` catches them, not only `check --references`. +// +// The two rules are ordered, not combined. An unqualified role has no module to +// compare, so the cross-module check would read the empty module as "some other +// module" and report a mismatch — a true-sounding message with the wrong +// diagnosis and advice that does not fix it. Qualification is therefore settled +// first, and a statement reported for MDL-GRANT02 is not tested for MDL-GRANT01. func ValidateGrantRoles(prog *ast.Program) []linter.Violation { var out []linter.Violation for _, stmt := range prog.Statements { + if v := validateRoleQualification(stmt); len(v) > 0 { + out = append(out, v...) + continue + } if err := validateCrossModuleGrant(stmt); err != nil { out = append(out, linter.Violation{ RuleID: "MDL-GRANT01", @@ -35,3 +52,74 @@ func ValidateGrantRoles(prog *ast.Program) []linter.Violation { } return out } + +// moduleRoleList returns the module roles a statement names, and a label for +// the statement to put in a message. +// +// Every statement built from the grammar's `moduleRoleList` belongs here. The +// entity grant is the one that made this a bug: MDL-GRANT01 was written for the +// five document grants, so `grant Wide on Sales.Order (...)` — the commonest +// GRANT of all — had no check-time coverage at all, and neither did the +// workflow grant or either user-role statement. +func moduleRoleList(stmt ast.Statement) (roles []ast.QualifiedName, what string) { + switch s := stmt.(type) { + case *ast.GrantEntityAccessStmt: + return s.Roles, "grant on " + s.Entity.String() + case *ast.GrantMicroflowAccessStmt: + return s.Roles, "grant execute on microflow " + s.Microflow.String() + case *ast.GrantNanoflowAccessStmt: + return s.Roles, "grant execute on nanoflow " + s.Nanoflow.String() + case *ast.GrantPageAccessStmt: + return s.Roles, "grant view on page " + s.Page.String() + case *ast.GrantWorkflowAccessStmt: + return s.Roles, "grant execute on workflow " + s.Workflow.String() + case *ast.GrantODataServiceAccessStmt: + return s.Roles, "grant access on OData service " + s.Service.String() + case *ast.GrantPublishedRestServiceAccessStmt: + return s.Roles, "grant access on published REST service " + s.Service.String() + case *ast.CreateUserRoleStmt: + return s.ModuleRoles, "create user role " + s.Name + case *ast.AlterUserRoleStmt: + return s.ModuleRoles, "alter user role " + s.Name + } + return nil, "" +} + +// validateRoleQualification reports (MDL-GRANT02) a module role written without +// its module. +// +// The grammar spells a module role as `qualifiedName`, whose module part is +// optional, so `Wide` parses as cleanly as `Sales.Wide` and reaches the executor +// with an empty Module. What happens next depends on the statement, and neither +// outcome is acceptable from a script that just passed `check`: +// +// - a GRANT fails at exec, by which point every earlier statement has already +// been written (mxcli does not run a script in one transaction); +// - `create user role R (Wide)` does not fail at all. It stores the reference +// as ".Wide" and reports success, and the defect surfaces only when mxbuild +// refuses the project with CE1613 "The selected module role '.Wide' no +// longer exists." +func validateRoleQualification(stmt ast.Statement) []linter.Violation { + roles, what := moduleRoleList(stmt) + if what == "" { + return nil + } + var out []linter.Violation + for _, role := range roles { + if role.Module != "" { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL-GRANT02", + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s names the module role %q without a module — a module role is always "+ + "Module.Role, and mxcli cannot tell which module %q belongs to", + what, role.Name, role.Name), + Suggestion: fmt.Sprintf( + "Qualify it, e.g. `.%s`. Run `show module roles` to list the roles a project has.", + role.Name), + }) + } + return out +} From e2d92f2bcd84c9a43bf179330981ea236fe894a4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 05:55:55 +0000 Subject: [PATCH 04/19] feat(entities): say which roles cannot see an attribute just added MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `alter entity E add attribute X` printed "Added attribute 'X'" and nothing else, including when it had just given that attribute to some roles at None. The field then renders blank for them, and every gate is green: check --references, mxbuild and `mx check` all pass (measured 0 errors on 11.13.0), because the model is not wrong — only narrower than the author expected. The report behind this described it as the attribute being added "only to access rules with the widest member lists, and silently skipped on narrower rules". That is not what happens, and it matters, because a fix aimed at the reported mechanism would have been wrong code for a real bug. Nothing is skipped: counted in the raw BSON, the new attribute's qualified name occurs exactly once per rule, which is why the build is clean. What differs is the RIGHTS. A new member joins each rule at that rule's DefaultMemberAccessRights, and MDL derives that property rather than setting it — `write *` gives ReadWrite, `read *` gives ReadOnly, and a grant written purely as member lists leaves it None. So the discriminator is the rule's default, not the width of its member list. The control that settles it, and that ships as the example: a rule granted `read *, write (Subject)` is narrower than `read *, write *` and still sees a newly added attribute, because `read *` set its default to ReadOnly. Only the member-listed rule is reported. The storage is therefore left alone — it matches what the property means — and what changes is the silence. The warning names the roles and prints the GRANT that widens the rule; verified that the suggested statement is additive and leaves the rule's existing members intact (`read (Subject)` becomes `read (Subject, Priority)`), so the advice does not quietly revoke anything. Two judgements worth knowing: - An EMPTY default counts as None. Mendix omits the property at its zero value, so reading empty as "sees new members" would silence the warning on exactly the rules it exists for. - A role named by ANY rule with a ReadOnly/ReadWrite default is not reported, even when another of its rules is member-listed, because Mendix combines the rights of every rule naming a role. The report is a set difference, not a per-rule scan. Scope is stated in the code rather than implied: the warning covers the entity's own access rules. A specialization carries its own rules over the inherited member and could be blind too, but its domain model is not held at this call and a generalization in another module is not reachable at all. Repro: mdl-examples/bug-tests/1067-new-member-access-warning.mdl Issue: mendixlabs/mxcli#1067 (c) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017s476QkXr9CFMvKspVzcvu --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../skills/mendix/manage-security/SKILL.md | 16 ++ cmd/mxcli/syntax/features_security.go | 11 +- docs-site/src/language/entity-access.md | 33 ++++ docs/01-project/MDL_QUICK_REFERENCE.md | 1 + .../1067-new-member-access-warning.mdl | 42 ++++++ mdl/executor/cmd_entities.go | 7 + .../cmd_entities_new_member_access.go | 103 +++++++++++++ .../cmd_entities_new_member_access_test.go | 142 ++++++++++++++++++ 9 files changed, 355 insertions(+), 1 deletion(-) create mode 100644 mdl-examples/bug-tests/1067-new-member-access-warning.mdl create mode 100644 mdl/executor/cmd_entities_new_member_access.go create mode 100644 mdl/executor/cmd_entities_new_member_access_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index cd8dc1f55f..77ae221147 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -567,3 +567,4 @@ {"area": "mdl/executor", "date": "2026-09-08", "refs": ["mendixlabs/mxcli#1073", "mendixlabs/mxcli#1020"], "ce": ["CE7252"], "symptom": "A `call external action` on an OData action that has a NULLABLE parameter is CE7252 \"The parameters for remote action '' have changed\", with no MDL that clears it. Reported as a missing syntax for an empty/null binding: `= null`, `= empty`, a bare `= )` and omitting the parameter were all tried. Persists after upgrading past the #1020 fix, because it is a different missing field behind the same CE code.", "file": "mdl/executor/cmd_microflows_builder_calls.go (externalParamKind.canBeEmpty, paramCanBeEmpty, resolveExternalActionParameterKinds, addCallExternalActionAction); engine-agnostic - the fix is in the shared semantic builder, so both modelsdk and legacy are covered by one change", "cause": "Microflows$ExternalActionParameterMapping.CanBeEmpty was never set, so it was Go's false on every mapping mxcli wrote. Mendix compares it against the contract's Nullable on every build and reports the disagreement as CE7252. mdl/types.EdmActionParameter ALREADY parsed Nullable as a three-state *bool; the value was simply dropped between the parser and the mapping builder, so the fix is to carry it - no new parsing. The default is the subtle half: CSDL makes Nullable optional on and defaults it to TRUE, the opposite of Go's zero value.", "insight": "**The reported premise was wrong and the bug was real; they were not the same thing.** A Studio Pro reference document settled both at once: on ako/TestApp 11.14.0 the mappings are {command, Argument \"empty\", CanBeEmpty false} and {additional, Argument \"empty\", CanBeEmpty true}. So (a) `Argument` is the EXPRESSION `empty`, never an empty string - an unfilled argument in Studio Pro is the Mendix null literal, so `additional = empty` was always correct MDL and already wrote a byte-identical Argument; and (b) the thing that actually differed was CanBeEmpty, which no MDL syntax reaches because it is derived from the contract, not typed by the developer. **Two hypotheses died on that one document**: that DESCRIBE's `additional = )` output (real - formatAction appends unconditionally where the java-action branch guards on empty) was what users hit, and that the empty-Argument state was reachable at all. It is not: DESCRIBE round-trips the real document correctly as `command = empty, additional = empty`. **Do not attribute a CE code to the last bug that produced it.** #1020 produced CE7252 from a missing ParameterType and was fixed in v0.21.0, which made \"upgrade\" look like the answer; the same code from a different field on HEAD was only found by building the reporter's shape and running mxbuild. **Verify version claims in the ISSUE against the grammar, not the changelog**: the three rules involved (callArgument, literal, callExternalActionStatement) are byte-identical at v0.20.0, so the reported parse errors for `= null`/`= empty` never happened at any version - an unrelated error elsewhere in the script (a missing microflow parameter list produces `missing '(' at 'begin'`) reads as an argument error and cost a probe here too. **Controls**: reverting only `mapping.CanBeEmpty = pk.canBeEmpty` takes the 4-statement repro from 0 to 4 errors, one CE7252 per call; separately, defaulting an ABSENT Nullable to false (rather than true) reproduces CE7252 on the Annotate action alone, which is what pins the CSDL default. Studio Pro's own microflow in the same project is the 0-error control. **Left unfixed on purpose**: Studio Pro also writes empty marker arrays AdditionalAttributes and IncludedAssociations (marker 2) on the call and each mapping; mxbuild 11.14 builds at 0 errors without them and they are unverified against Studio Pro, so they are recorded rather than guessed at. Repro mdl-examples/bug-tests/odata-1073-external-action-nullable-params.mdl"} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "CI build-and-test fails on a newly added mdl-examples/doctype-tests/ script with `Execution error: ... needs the modelsdk engine (run without MXCLI_ENGINE=legacy)` — while `mxcli check` and a local exec both pass", "cause": "TestMxCheck_DoctypeScripts runs every doctype script through exec + mx check on BOTH engines. A script using a modelsdk-only capability (creating a navigation profile, menu/rule/layout authoring) cannot pass on legacy, where the backend refuses by design rather than approximating the document", "file": "`mdl/executor/roundtrip_doctype_test.go` (engineScriptSkip)", "insight": "A doctype example is a two-engine test, not a one-engine one, and nothing local tells you: `mxcli check` needs no engine and a local exec uses the default (modelsdk). Before adding an example, ask whether anything in it is modelsdk-only — the refusals are deliberate and listed in mdl/backend/mpr/backend.go. The remedy is an engineScriptSkip entry naming WHY the engine refuses, not weakening the script; and note separately whether the feature under test is itself dual-engine, since here only the profile CREATION was modelsdk-only while the SYNC block works on both and is unit-tested on each", "refs": ["ako/mxcli#420"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A module role written WITHOUT its module (`grant Wide on Sales.Order (read *)`) passes both `mxcli check` and `check --references`. The GRANT forms then fail at exec — after the preceding statements have been written — with the malformed message `failed to module not found for role .Wide`. `create user role R (Wide)` does not fail at all: it reports success and stores the reference as \".Wide\", and only MxBuild refuses the project, with CE1613 \"The selected module role '.Wide' no longer exists.\"", "cause": "The grammar spells a module role as `qualifiedName`, whose module part is optional (`identifierOrKeyword (DOT identifierOrKeyword)*`), so a bare name parses and reaches the executor with an empty Module. MDL-GRANT01 was the only check-time rule reading role lists, and it was written for the five DOCUMENT grants: `grant ... on `, the workflow grant and both user-role statements were never in its switch. On the document grants it did fire, but with the wrong diagnosis — it compared the empty module against the document's and reported a CROSS-MODULE error (CE0148), advice that does not fix a missing qualifier. The user-role path never validated at all; it concatenated `mr.Module + \".\" + mr.Name` and stored the result.", "file": "`mdl/executor/validate_grant_roles.go` (MDL-GRANT02, `moduleRoleList` + `validateRoleQualification`), `mdl/executor/cmd_security_write.go` (`qualifiedModuleRoleNames`, `validateModuleRole`)", "insight": "**Two switches over the same statement set, with nothing comparing them** — the same shape as the check-coverage defect fixed in `validate_duplicates.go` two days earlier (stmtCreateInfo 24 types vs setFor 20). Here it is the grammar's `moduleRoleList` (9 statements) against MDL-GRANT01's switch (5). When a rule reads a grammar list, enumerate the list, not the statements you happened to think of. **Order the two rules rather than merging them**: an unqualified role has no module to compare, so the cross-module check reads empty as \"some other module\" and produces a true-sounding message with the wrong remedy — qualification is settled first and suppresses the second rule for that statement. **A late error and a stored corruption are not the same severity.** The GRANT forms failed loudly at exec; the user-role form succeeded and wrote a dangling reference, which is worse and was found only by running the statement and then `mx check`. When auditing a validation gap, run each affected statement to the end rather than assuming they all fail the same way. Guard both layers: the pre-flight makes `exec` refuse before writing anything, and the executor guard still fires on the `-c` path, which skips the pre-flight — that is the control proving the fix is not only in the checker. Repros `mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl`. Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067", "#836"], "ce": ["CE1613", "CE0148"], "rules": ["MDL-GRANT02"]} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "After `alter entity E add attribute X`, the new attribute is invisible to some roles — it renders blank in the UI — while `check --references`, `mxbuild` and `mx check` all report the model clean (measured: 0 errors on 11.13.0). Reported as the attribute being \"added only to access rules with the widest member lists, and silently skipped on narrower rules\".", "cause": "Nothing is skipped. Measured on the raw BSON, the new attribute's qualified name occurs exactly ONCE PER RULE — every access rule gets its MemberAccess, which is why the model is structurally complete and builds clean. What differs is the RIGHTS: a new member joins each rule at that rule's `DefaultMemberAccessRights`, and MDL derives that property instead of setting it — `write *` gives ReadWrite, `read *` gives ReadOnly, and a grant written purely as member lists (`grant R on E (read (Subject))`) leaves it None. So the storage is right and matches what the property means; the defect was that `alter entity` printed only \"Added attribute 'X'\" and said nothing about the roles that had just been given None.", "file": "`mdl/executor/cmd_entities_new_member_access.go` (`rolesBlindToNewMembers`, `newMemberAccessWarning`), wired at the `ast.AlterEntityAddAttribute` branch of `mdl/executor/cmd_entities.go`", "insight": "**The reporter's mechanism was wrong and the symptom was right — take the symptom and re-derive the mechanism.** The control that settles it: a third rule granted `read *, write (Subject)` is NARROWER than `read *, write *` and still sees a newly added attribute, because `read *` set its default to ReadOnly. So the discriminator is the rule's default, not the width of its member list — and a fix aimed at \"copy the widest rule's members\" would have been wrong code for a real bug. **Count the BSON before believing a \"silently skipped\" report**: `grep -oa 'Mod.Ent.Attr' .mxunit | wc -l` distinguished \"entry missing\" (would be CE0066) from \"entry present at None\" (clean build) in one command, and they call for opposite fixes. A role named by ANY rule with a ReadOnly/ReadWrite default is not blind even if another of its rules is member-listed — Mendix combines the rights of every rule naming a role — so the report is a set difference, not a per-rule scan. Treat an EMPTY default as None: Mendix omits the property at its zero value, and reading empty as \"sees new members\" silences the warning on exactly the rules it exists for. Scope limit recorded in the code: the warning covers the entity's own rules, not a specialization's, because the descendants live in domain models the call does not hold. Repro `mdl-examples/bug-tests/1067-new-member-access-warning.mdl`. Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067", "#936"], "ce": ["CE0066"]} diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 07f64b8ce6..92908590a8 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -229,6 +229,22 @@ revoke MyModule.User on MyModule.Customer (write (Email)); revoke MyModule.User on MyModule.Customer (delete); ``` +#### Members added later + +A rule also carries a default for members added **after** it was written, and MDL +derives it from the grant: `write *` → ReadWrite, `read *` → ReadOnly, and a +grant written **purely as member lists** leaves it at **None**. + +So `alter entity … add attribute` gives the new attribute None on a +member-listed rule. Nothing is broken — every rule gets an entry, the build is +clean — but the attribute renders blank for that role. `alter entity` warns and +prints the grant that widens it. + +What decides this is the rule's default, **not** how narrow its member list is: +`read *, write (Email)` is narrower than `read *, write *` and still picks up new +members, because `read *` set its default to ReadOnly. Give a rule `read *` and +narrow with `revoke` when the role should follow the entity as it grows. + #### Inherited members Mendix inheritance is multi-table: a child adds attributes to its parent's, and diff --git a/cmd/mxcli/syntax/features_security.go b/cmd/mxcli/syntax/features_security.go index 0c59fbece4..74b0ea1a41 100644 --- a/cmd/mxcli/syntax/features_security.go +++ b/cmd/mxcli/syntax/features_security.go @@ -38,7 +38,16 @@ func init() { "REVOKE . ON . ();\n\n" + "Rights: CREATE, DELETE, READ *, READ (,...), WRITE *, WRITE (,...)\n\n" + "A module role is always Module.Role. A bare role name parses but is\n" + - "refused (MDL-GRANT02) \u2014 mxcli cannot tell which module it belongs to.\n\n" + + "refused (MDL-GRANT02) — mxcli cannot tell which module it belongs to.\n\n" + + "Members added later:\n" + + " A rule also carries a default for members added AFTER it was written,\n" + + " derived from the grant: WRITE * gives ReadWrite, READ * gives ReadOnly,\n" + + " and member lists alone leave it None. So an attribute added later is\n" + + " granted None on a member-listed rule — a clean build in which the field\n" + + " renders blank for that role. ALTER ENTITY ... ADD ATTRIBUTE warns and\n" + + " prints the GRANT that widens it. What decides this is the rule's\n" + + " default, not how narrow its member list is: READ *, WRITE (Email) is\n" + + " narrower than READ *, WRITE * and still picks up new members.\n\n" + "Inherited members:\n" + " Mendix inheritance is multi-table — a child adds attributes to its\n" + " parent's, and ALL the parent's members belong to the child. Name them\n" + diff --git a/docs-site/src/language/entity-access.md b/docs-site/src/language/entity-access.md index 0249fc2851..b6391d178c 100644 --- a/docs-site/src/language/entity-access.md +++ b/docs-site/src/language/entity-access.md @@ -43,6 +43,39 @@ Restrict read and write to specific attributes: GRANT Shop.User ON Shop.Customer (READ (Name, Email, Status), WRITE (Email)); ``` +### Members Added Later + +A rule also carries a **default for members added after it was written**, and MDL +derives that default from the grant rather than stating it: `WRITE *` gives +ReadWrite, `READ *` gives ReadOnly, and a grant written **purely as member +lists** leaves it at **None**. + +So an attribute added later is granted None on a member-listed rule. The model +is complete and correct — every rule gets an entry for the new member, and the +build reports no errors — but the attribute renders blank for that role: + +```sql +GRANT Shop.User ON Shop.Customer (READ (Name, Email)); +-- later: +-- alter entity Shop.Customer add attribute Phone: String; +-- Phone is granted None to Shop.User. Nothing is broken; it is simply not visible. +``` + +`alter entity … add attribute` reports this and prints the GRANT that fixes it: + +``` +Added attribute 'Phone' to entity Shop.Customer +Warning: Shop.Customer.Phone is not readable by Shop.User + ... + grant Shop.User on Shop.Customer (read (Phone)); +``` + +What decides this is the rule's **default**, not how narrow its member list is. A +rule granted `READ *, WRITE (Email)` is narrower than `READ *, WRITE *` and still +sees new members, because `READ *` set its default to ReadOnly. If you want a +role to pick up future members automatically, give its rule a `READ *` or +`WRITE *` and narrow from there with [REVOKE](#revoke-on-entities). + ### XPath Constraints Limit which objects a role can see or modify using an XPath expression in the `WHERE` clause: diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index daa734aedd..d0ba3dcd8f 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -626,6 +626,7 @@ Nested folders use `/` separator: `'Parent/Child/Grandchild'`. Missing folders a | Grant page access | `grant view on page Mod.Page to Mod.Role, ...;` | | | Revoke page access | `revoke view on page Mod.Page from Mod.Role, ...;` | | | Grant entity access | `grant Mod.Role on Mod.Entity (create, delete, read *, write *);` | Additive — merges with existing. A module role must be qualified: a bare `Role` parses but is refused (MDL-GRANT02). Inherited members are named like the entity's own (`read *` covers them); an unknown name is an error. Entities extending `System.User` are the exception — their platform members must not be granted | +| Access for members added later | — | A rule's default for new members is derived from the grant: `write *` → ReadWrite, `read *` → ReadOnly, member lists alone → **None**. So an attribute added later is granted None on a member-listed rule — clean build, blank field. `alter entity … add attribute` warns and prints the widening grant. The rule's *default* decides this, not how narrow its member list is | | Revoke entity access | `revoke Mod.Role on Mod.Entity;` | Full revoke — removes entire rule | | Revoke entity access (partial) | `revoke Mod.Role on Mod.Entity (read (attr));` | Partial — downgrades specific rights | | Set security level | `alter project security level off\|prototype\|production;` | | diff --git a/mdl-examples/bug-tests/1067-new-member-access-warning.mdl b/mdl-examples/bug-tests/1067-new-member-access-warning.mdl new file mode 100644 index 0000000000..854dc7a877 --- /dev/null +++ b/mdl-examples/bug-tests/1067-new-member-access-warning.mdl @@ -0,0 +1,42 @@ +-- mendixlabs/mxcli#1067 (c): a member added later is invisible to a role whose +-- access rule was granted per member. +-- +-- Run this, then add an attribute: +-- +-- mxcli exec 1067-new-member-access-warning.mdl -p app.mpr +-- mxcli -p app.mpr -c "alter entity Zz1067c.Doc add attribute Priority: String" +-- +-- `alter entity` used to print only "Added attribute 'Priority'". It now also +-- reports that Zz1067c.Narrow cannot see it, and prints the GRANT that widens +-- the rule. +-- +-- Why: a new member joins each access rule at that rule's +-- DefaultMemberAccessRights. MDL derives that property rather than setting it — +-- `write *` gives ReadWrite, `read *` gives ReadOnly, and a grant written purely +-- as a member list leaves it at None. Nothing is skipped: measured on 11.13.0, +-- the BSON carries one MemberAccess per rule for the new attribute and mxbuild +-- reports 0 errors. The attribute is simply granted None, so it renders blank. +-- +-- Note which rule is NOT reported. Zz1067c.Mid is narrower than Zz1067c.Wide — +-- it can read everything but write only one member — and it still sees new +-- members, because `read *` set its default to ReadOnly. The discriminator is +-- the rule's default, not the width of its member list. + +create module Zz1067c; +create module role Zz1067c.Wide; +create module role Zz1067c.Mid; +create module role Zz1067c.Narrow; + +create entity Zz1067c.Doc ( + Subject: String, + Body: String +); + +-- Default ReadWrite: sees new members. +grant Zz1067c.Wide on Zz1067c.Doc (create, delete, read *, write *); + +-- Default ReadOnly: sees new members, despite the narrower write list. +grant Zz1067c.Mid on Zz1067c.Doc (read *, write (Subject)); + +-- Default None: does NOT see new members. This is the one reported. +grant Zz1067c.Narrow on Zz1067c.Doc (read (Subject)); diff --git a/mdl/executor/cmd_entities.go b/mdl/executor/cmd_entities.go index 7df2377fba..142e4fb6ca 100644 --- a/mdl/executor/cmd_entities.go +++ b/mdl/executor/cmd_entities.go @@ -1012,6 +1012,13 @@ func execAlterEntity(ctx *ExecContext, s *ast.AlterEntityStmt) error { invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) fmt.Fprintf(ctx.Output, "Added attribute '%s' to entity %s\n", a.Name, s.Name) + // The new attribute joins each access rule at that rule's default member + // rights, so a rule granted per-member gets it at None and the roles on + // that rule cannot see it. Valid model, clean build, blank field + // (mendixlabs/mxcli#1067) — say so rather than leaving it to runtime. + if w := newMemberAccessWarning(s.Name.String(), a.Name, rolesBlindToNewMembers(entity)); w != "" { + fmt.Fprint(ctx.Output, w) + } case ast.AlterEntityRenameAttribute: var target *domainmodel.Attribute diff --git a/mdl/executor/cmd_entities_new_member_access.go b/mdl/executor/cmd_entities_new_member_access.go new file mode 100644 index 0000000000..f4db8717a1 --- /dev/null +++ b/mdl/executor/cmd_entities_new_member_access.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Reporting for the access a newly added entity member does NOT get. +// +// Mendix gives a new member the rights named by its access rule's +// DefaultMemberAccessRights — that property is the rule's answer to "what should +// a member added later be allowed to do". MDL cannot set it directly: it is +// derived from the grant, `write *` → ReadWrite, `read *` → ReadOnly, and a +// grant written purely as a member list (`grant R on E (read (Subject))`) leaves +// it at None. +// +// So `alter entity … add attribute` on an entity carrying such a rule produces a +// model that is structurally complete — every rule gets a MemberAccess for the +// new member, and mxbuild reports 0 errors — in which the new attribute is +// nonetheless invisible to those roles. It renders blank in the UI for them and +// nothing in the toolchain says why (mendixlabs/mxcli#1067). +// +// The storage is not wrong; the silence is. This prints what the write implied. +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// rolesBlindToNewMembers returns the module roles that will not see a member +// added to this entity now: the roles named ONLY by access rules whose default +// member rights are None. Sorted and de-duplicated. +// +// A role named by any rule with ReadOnly or ReadWrite defaults is excluded even +// when another of its rules is member-listed, because Mendix combines the access +// rights of every rule naming a role — the reference guide is explicit that +// several rules may name the same module role and that "all access rights of +// those rules are combined". Reporting such a role would be a false positive. +// +// Scope: this entity's own access rules. A specialization of this entity carries +// its own rules over the inherited member and could be blind too, but resolving +// the descendants means walking domain models this call does not hold — and a +// generalization in another module is not reachable at all. Widening the report +// there is worth doing separately; claiming it here without doing it would be +// worse than the stated limit. +func rolesBlindToNewMembers(entity *domainmodel.Entity) []string { + if entity == nil { + return nil + } + blind := map[string]bool{} + sighted := map[string]bool{} + for _, rule := range entity.AccessRules { + if rule == nil { + continue + } + // An empty default is None: Mendix's own documents leave the property off + // when it carries the zero value, and reading empty as "sees new members" + // would silence the warning on the very rules it exists for. + seesNewMembers := rule.DefaultMemberAccessRights == domainmodel.MemberAccessRightsReadOnly || + rule.DefaultMemberAccessRights == domainmodel.MemberAccessRightsReadWrite + for _, role := range rule.ModuleRoleNames { + if role == "" { + continue + } + if seesNewMembers { + sighted[role] = true + } else { + blind[role] = true + } + } + } + + var out []string + for role := range blind { + if !sighted[role] { + out = append(out, role) + } + } + sort.Strings(out) + return out +} + +// newMemberAccessWarning renders the notice for a member just added to +// entityQName, or "" when every role can see it. +// +// It names the remedy as a statement the reader can paste. GRANT is additive — +// it widens a rule and never narrows (narrowing is REVOKE's job) — so re-granting +// the one member is the minimal, non-destructive fix and leaves the rule's other +// members alone. +func newMemberAccessWarning(entityQName, memberName string, blindRoles []string) string { + if len(blindRoles) == 0 { + return "" + } + var b strings.Builder + fmt.Fprintf(&b, "Warning: %s.%s is not readable by %s\n", + entityQName, memberName, strings.Join(blindRoles, ", ")) + fmt.Fprintf(&b, " Those access rules grant rights per member, so a member added later joins them\n") + fmt.Fprintf(&b, " with no access — the model is valid and builds clean, but the attribute renders\n") + fmt.Fprintf(&b, " blank for those roles. Widen a rule with:\n") + for _, role := range blindRoles { + fmt.Fprintf(&b, " grant %s on %s (read (%s));\n", role, entityQName, memberName) + } + return b.String() +} diff --git a/mdl/executor/cmd_entities_new_member_access_test.go b/mdl/executor/cmd_entities_new_member_access_test.go new file mode 100644 index 0000000000..54211a4d1f --- /dev/null +++ b/mdl/executor/cmd_entities_new_member_access_test.go @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mendixlabs/mxcli#1067 (c): `alter entity … add attribute` reported only +// "Added attribute 'X'", and said nothing about the roles that could not see it. +// +// The reporter read the result as the attribute being "added only to access +// rules with the widest member lists, silently skipped on narrower rules". It is +// not skipped anywhere — measured on an 11.13.0 project, the raw BSON carries +// exactly one MemberAccess per rule for the new attribute, and mxbuild reports 0 +// errors. What differs is the RIGHTS: a new member joins each rule at that +// rule's DefaultMemberAccessRights, and a grant written as a member list +// (`grant R on E (read (Subject))`) leaves that default at None. +// +// So the discriminator is the rule's default, not the width of its member list. +// The control that settles it: a rule granted `read *, write (Subject)` — a +// NARROWER rule than `read *, write *` — still sees a newly added attribute, +// because `read *` sets its default to ReadOnly. +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +func ruleWithDefault(def domainmodel.MemberAccessRights, roles ...string) *domainmodel.AccessRule { + return &domainmodel.AccessRule{ + ModuleRoleNames: roles, + DefaultMemberAccessRights: def, + } +} + +func TestRolesBlindToNewMembers(t *testing.T) { + cases := []struct { + name string + rules []*domainmodel.AccessRule + want []string + }{ + { + name: "no access rules at all", + rules: nil, + want: nil, + }, + { + // `grant R on E (create, delete, read *, write *)` + name: "ReadWrite default sees new members", + rules: []*domainmodel.AccessRule{ruleWithDefault(domainmodel.MemberAccessRightsReadWrite, "Sales.Wide")}, + want: nil, + }, + { + // `grant R on E (read *, write (Subject))` — the control. A narrower + // rule than the one above, and still not blind. + name: "ReadOnly default sees new members", + rules: []*domainmodel.AccessRule{ruleWithDefault(domainmodel.MemberAccessRightsReadOnly, "Sales.Mid")}, + want: nil, + }, + { + // `grant R on E (read (Subject))` + name: "None default is blind", + rules: []*domainmodel.AccessRule{ruleWithDefault(domainmodel.MemberAccessRightsNone, "Sales.Narrow")}, + want: []string{"Sales.Narrow"}, + }, + { + // An access rule Studio Pro wrote may leave the property empty rather + // than spelling "None". Treating empty as "not blind" would make the + // warning silent on exactly the rules it exists for. + name: "empty default is None", + rules: []*domainmodel.AccessRule{ruleWithDefault("", "Sales.Blank")}, + want: []string{"Sales.Blank"}, + }, + { + name: "only the blind rules are named", + rules: []*domainmodel.AccessRule{ + ruleWithDefault(domainmodel.MemberAccessRightsReadWrite, "Sales.Wide"), + ruleWithDefault(domainmodel.MemberAccessRightsNone, "Sales.Narrow"), + ruleWithDefault(domainmodel.MemberAccessRightsReadOnly, "Sales.Mid"), + }, + want: []string{"Sales.Narrow"}, + }, + { + // One rule may name several roles, and several rules may name the + // same role. The report is a set, sorted, so it is stable. + name: "roles are de-duplicated and sorted", + rules: []*domainmodel.AccessRule{ + ruleWithDefault(domainmodel.MemberAccessRightsNone, "Sales.Zeta", "Sales.Alpha"), + ruleWithDefault(domainmodel.MemberAccessRightsNone, "Sales.Alpha"), + }, + want: []string{"Sales.Alpha", "Sales.Zeta"}, + }, + { + // A role covered by ANY rule that sees new members is not blind, even + // if another of its rules is member-listed. Mendix combines the access + // rights of every rule naming a role, so reporting it would be wrong. + name: "a role with a second, sighted rule is not blind", + rules: []*domainmodel.AccessRule{ + ruleWithDefault(domainmodel.MemberAccessRightsNone, "Sales.Both"), + ruleWithDefault(domainmodel.MemberAccessRightsReadOnly, "Sales.Both"), + }, + want: nil, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := rolesBlindToNewMembers(&domainmodel.Entity{AccessRules: tc.rules}) + if len(got) != len(tc.want) { + t.Fatalf("got %v, want %v", got, tc.want) + } + for i := range got { + if got[i] != tc.want[i] { + t.Fatalf("got %v, want %v", got, tc.want) + } + } + }) + } +} + +// The warning has to be actionable: it must name the attribute, the roles, and +// the statement that widens the rule. A warning that only says "some roles +// cannot see this" costs the reader the same investigation it was meant to save. +func TestNewMemberAccessWarning_IsActionable(t *testing.T) { + got := newMemberAccessWarning("Sales.Order", "Priority", []string{"Sales.Narrow", "Sales.Other"}) + for _, want := range []string{ + "Priority", + "Sales.Narrow", + "Sales.Other", + "grant Sales.Narrow on Sales.Order (read (Priority))", + } { + if !strings.Contains(got, want) { + t.Errorf("warning must contain %q, got:\n%s", want, got) + } + } +} + +// No blind roles means no output. A warning printed on every ADD ATTRIBUTE is a +// warning nobody reads. +func TestNewMemberAccessWarning_SilentWhenNothingIsBlind(t *testing.T) { + if got := newMemberAccessWarning("Sales.Order", "Priority", nil); got != "" { + t.Errorf("want empty, got: %q", got) + } +} From 39436a60bcdf669dc0d0c5e70a6ba6bb94276356 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:03:30 +0000 Subject: [PATCH 05/19] fix(widgets): lift the editor hide-rules describe was missing, conjunctions included MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mxcli widget describe` over-listed the bindings a widget requires, because half of each widget's editor visibility rules were never lifted from its editorConfig.js. Combo box reported "16 of 32 editor hide-rules recognized", so properties its editor hides in the described configuration were still asked for. Across the 25 fixture widgets carrying rules: 99 of 177 hide-calls, 56%. Two independent gaps. The guard vocabulary covered string equality and bare truthiness only, so `null===e.dataSource` ("no datasource picked"), terser's minified booleans `!1===e.showFooter`, `0===e.list.length` and `["a","b"].includes(e.type)` all read as unsupported. `null` and `0===x.length` get their own `empty` operator rather than reusing `falsy`: "false" and "0" are values a real property holds, and treating them as unset hides a property the author is using. A hide that was not FIRST in a `cond ? (hide(a), hide(b))` comma group also got no guard at all. The rule model held ONE condition, but editorConfig nests its branches — Combo box reaches `source=="context" && optionsSourceType=="association" && showFooter==false` three levels deep. A rule keeping only the innermost term claims hidden in configurations the editor shows: read that way, Datagrid's `pagingPosition` is hidden whenever the row count is off, pagination or not, and pagination defaults on. Rules now carry the whole conjunction, and every consumer evaluates it through Rule.Fires — a reader that looks only at HiddenWhen silently over-fires, which is the same bug wearing the right field name. One indeterminable term makes the rule indeterminable, so nothing is pruned and the binding is still listed. Three sub-traps, each found by diffing extracted rules against the editorConfig source rather than by reasoning: the innermost enclosing group is often the very group whose guard the hide already carries, so terms are deduped; the comma walk must not cross a comma that is not a group separator, which had given Maps' `advanced` a guard belonging to a different property; and the platform argument is not a configuration term — TreeNode gates on `"web"===platform`, which every page MDL writes satisfies, so folding it away keeps three bindings pruned. Measured over the fixture's 33 widgets: 99 -> 116 of 177 recognized (56% -> 66%), zero rules lost, zero per-widget regressions, and exactly one binding removed from a generated example (TreeNode's `openNodeOn`, traced to `"text"===e.headerType ? (hide("headerContent"), hide("openNodeOn"))`). Combo box's example drops from 19 bindings to 18. Controls: removing the new guard shapes, the comma-group attribution, the stored conjunction terms, or the consumers' evaluation of them each makes the matching test fail with the reported symptom. Coverage alone was a misleading target and the first cut proved it — it took Combo box to 27 of 32 and left the describe output byte-identical, because consumers skip an indeterminable condition and `exampleValues` records a property only when its default is non-empty, which an unset datasource never is. Several of those extra rules were also simply wrong. The acceptance bar here is zero rules lost and every removed binding traced to the source, not a larger number. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KTdvGwVewkdgNQGDCxpQPZ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../src/reference/query/describe-widget.md | 15 + mdl/backend/widgetobj/builder.go | 8 +- mdl/executor/editorconfig_extract.go | 677 +++++++++++++++++- mdl/executor/editorconfig_shapes_test.go | 243 +++++++ mdl/executor/validate_widget_hidden.go | 20 +- mdl/executor/validate_widgets.go | 14 +- mdl/executor/widget_describe.go | 88 ++- mdl/executor/widget_engine.go | 32 +- mdl/types/widget_visibility.go | 91 ++- 10 files changed, 1152 insertions(+), 37 deletions(-) create mode 100644 mdl/executor/editorconfig_shapes_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index b4c3b7aa14..bdfa6018d8 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -563,3 +563,4 @@ {"area": "mdl/executor", "date": "2026-09-08", "symptom": "`mxcli check --references` reported EVERY enumeration as missing — `attribute 'CriticalPathStation': enumeration not found: Approval.StationKey` — while `DESCRIBE ENUMERATION` returned its values, `SHOW ENUMERATIONS` listed it, `exec` wrote the attribute and mxbuild built the project at 0 errors (mendixlabs/mxcli#1071). A pure false negative: the only broken thing was the checker.", "cause": "`enumerationExists` (mdl/executor/helpers.go) matched containers directly — `enum.ContainerID == module.ID` — which only holds for an enumeration sitting in the module ROOT; one inside a FOLDER has the folder as its container. Every other command resolves through the container hierarchy (`h.GetModuleName(h.FindModuleID(e.ContainerID))`), so the reference checker was the only one that could not see inside a folder. Fixed by deferring to `findEnumeration`, deleting the duplicate rather than patching the copy.", "file": "`mdl/executor/helpers.go` (enumerationExists); call sites `mdl/executor/validate.go:447` (CREATE ENTITY) and `:660` (ALTER ENTITY ADD ATTRIBUTE); tests `mdl/executor/validate_enum_folder_test.go`; example `mdl-examples/bug-tests/1071-foldered-enum-references.mdl`", "insight": "This is upstream #976 a second time. That fix corrected DROP's container matching and did NOT sweep for the other callers asking the same question, so the identical bug sat in the reference checker for months — and its own test file already spelled out the class (\"SHOW, DESCRIBE and ALTER all use the container hierarchy... DROP was the one command of the four\"). When a fix is 'this command resolved containers wrongly', grep for every other place that resolves the same containers before closing it; the enumeration existed in TWO implementations and only the interactive one was ever exercised. Two measurement notes: the report read as 'enumerations are never resolved' because the reporter's module keeps them in folders, so the discriminator (root vs foldered, same module, same script) had to be built before anything else made sense; and the blast radius was larger than reported — CREATE ENTITY fails identically and the report only showed ALTER, so the test covers both call sites.", "ce": []} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "`describe navigation` on a project with an offline profile emits a broken comment: the line ends after `where '[` and several lines of raw XPath follow as if they were MDL", "cause": "The offline-entity comment interpolated the stored constraint verbatim. Studio Pro writes an offline sync constraint multi-line, with indentation and Mendix's doubled-quote escaping, and a newline inside a `--` comment ends the comment", "file": "`mdl/executor/cmd_navigation.go` (singleLine)", "insight": "A value that is single-line in every fixture can be multi-line in every real document. This survived because no local project had a populated offline config at all — the constraint had never been rendered with real content. When a describe path interpolates a stored string into a line-oriented format, fold it; the fixture that would have caught this is one carrying a value copied out of a real project rather than typed into a test", "refs": ["ako/TestApp", "PROPOSAL_offline_sync_configuration.md"]} {"area": "mdl/executor", "date": "2026-09-08", "refs": ["mendixlabs/mxcli#1073", "mendixlabs/mxcli#1020"], "ce": ["CE7252"], "symptom": "A `call external action` on an OData action that has a NULLABLE parameter is CE7252 \"The parameters for remote action '' have changed\", with no MDL that clears it. Reported as a missing syntax for an empty/null binding: `= null`, `= empty`, a bare `= )` and omitting the parameter were all tried. Persists after upgrading past the #1020 fix, because it is a different missing field behind the same CE code.", "file": "mdl/executor/cmd_microflows_builder_calls.go (externalParamKind.canBeEmpty, paramCanBeEmpty, resolveExternalActionParameterKinds, addCallExternalActionAction); engine-agnostic - the fix is in the shared semantic builder, so both modelsdk and legacy are covered by one change", "cause": "Microflows$ExternalActionParameterMapping.CanBeEmpty was never set, so it was Go's false on every mapping mxcli wrote. Mendix compares it against the contract's Nullable on every build and reports the disagreement as CE7252. mdl/types.EdmActionParameter ALREADY parsed Nullable as a three-state *bool; the value was simply dropped between the parser and the mapping builder, so the fix is to carry it - no new parsing. The default is the subtle half: CSDL makes Nullable optional on and defaults it to TRUE, the opposite of Go's zero value.", "insight": "**The reported premise was wrong and the bug was real; they were not the same thing.** A Studio Pro reference document settled both at once: on ako/TestApp 11.14.0 the mappings are {command, Argument \"empty\", CanBeEmpty false} and {additional, Argument \"empty\", CanBeEmpty true}. So (a) `Argument` is the EXPRESSION `empty`, never an empty string - an unfilled argument in Studio Pro is the Mendix null literal, so `additional = empty` was always correct MDL and already wrote a byte-identical Argument; and (b) the thing that actually differed was CanBeEmpty, which no MDL syntax reaches because it is derived from the contract, not typed by the developer. **Two hypotheses died on that one document**: that DESCRIBE's `additional = )` output (real - formatAction appends unconditionally where the java-action branch guards on empty) was what users hit, and that the empty-Argument state was reachable at all. It is not: DESCRIBE round-trips the real document correctly as `command = empty, additional = empty`. **Do not attribute a CE code to the last bug that produced it.** #1020 produced CE7252 from a missing ParameterType and was fixed in v0.21.0, which made \"upgrade\" look like the answer; the same code from a different field on HEAD was only found by building the reporter's shape and running mxbuild. **Verify version claims in the ISSUE against the grammar, not the changelog**: the three rules involved (callArgument, literal, callExternalActionStatement) are byte-identical at v0.20.0, so the reported parse errors for `= null`/`= empty` never happened at any version - an unrelated error elsewhere in the script (a missing microflow parameter list produces `missing '(' at 'begin'`) reads as an argument error and cost a probe here too. **Controls**: reverting only `mapping.CanBeEmpty = pk.canBeEmpty` takes the 4-statement repro from 0 to 4 errors, one CE7252 per call; separately, defaulting an ABSENT Nullable to false (rather than true) reproduces CE7252 on the Annotate action alone, which is what pins the CSDL default. Studio Pro's own microflow in the same project is the 0-error control. **Left unfixed on purpose**: Studio Pro also writes empty marker arrays AdditionalAttributes and IncludedAssociations (marker 2) on the call and each mapping; mxbuild 11.14 builds at 0 errors without them and they are unverified against Studio Pro, so they are recorded rather than guessed at. Repro mdl-examples/bug-tests/odata-1073-external-action-nullable-params.mdl"} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "`mxcli widget describe` over-listed required bindings because half of each widget's editor visibility rules were never lifted: Combo box reported '16 of 32 editor hide-rules recognized', so properties its editor hides in the described configuration were still asked for. Across the 25 widgets in the fixture that carry rules, 99 of 177 hide-calls (56%) were recognized.", "cause": "Two independent gaps. (1) The guard vocabulary covered only string equality and bare truthiness, so `null===e.dataSource`, minified booleans `!1===e.showFooter`, `0===e.list.length` and `[\"a\",\"b\"].includes(e.type)` all read as unsupported; and a hide that was not FIRST in a `cond ? (hide(a), hide(b))` comma group got no guard at all. (2) The rule model held ONE condition, but editorConfig nests branches — Combo box reaches `source==\"context\" && optionsSourceType==\"association\" && showFooter==false` three levels deep — so a lifted rule could only ever be one conjunct.", "file": "mdl/executor/editorconfig_extract.go, mdl/types/widget_visibility.go", "insight": "Coverage is the wrong thing to optimise on its own: the first cut took Combo box 16->27 and left the describe output BYTE-IDENTICAL, because every consumer skips a rule whose condition value is indeterminable and `exampleValues` records a property only when its default is non-empty — which an unset datasource never is. Moving a metric without moving the outcome is the failure mode to watch for; measure the user-visible artifact (the generated MDL example), not the counter. Worse, several of those 11 new rules were WRONG: they stated one conjunct of a nested condition, and `pagingPosition` (Datagrid) then read as hidden whenever the row count is off, pagination or not — pagination defaults ON, so that is the common case. The fix is to carry the whole conjunction (WidgetVisibilityRule.And) rather than to reject or to guess, and to make every consumer evaluate it (Rule.Fires) — a reader that looks only at HiddenWhen silently over-fires, which is exactly the bug wearing the right field name. Three sub-traps, each found only by diffing rules against the editorConfig source: the innermost enclosing group is often the very group whose guard the hide already carries, so terms must be deduped; the comma walk must not cross a comma that is not a group separator (`A?B:hide(x),hide(y)` gave Maps' `advanced` a guard belonging to a different property); and the PLATFORM argument is not a configuration term — TreeNode gates on `\"web\"===platform`, which MDL always satisfies, so folding it away keeps three bindings correctly pruned. Discipline that made this safe: the acceptance bar was 'zero rules lost, zero regressions, and every removed binding traced to the editorConfig source', not 'the number went up'.", "refs": ["mendixlabs/mxcli#1036"]} diff --git a/docs-site/src/reference/query/describe-widget.md b/docs-site/src/reference/query/describe-widget.md index 969d7be7c4..38cf9247f9 100644 --- a/docs-site/src/reference/query/describe-widget.md +++ b/docs-site/src/reference/query/describe-widget.md @@ -103,6 +103,21 @@ widget hides under the configuration the example picked is left out, so what you see is what that configuration actually supports. The footer reports how many of the widget's hide-rules were recognised; an unrecognised rule never prunes. +A rule can carry several conditions, joined with `and`: + +``` +clearable hidden when optionsSourceType = "boolean" + and optionsSourceType is one of enumeration, boolean + and source = "context" +``` + +That is not verbosity — it is the rule. Widget editors nest their branches, and +a property hidden three levels in is hidden only where **all** three hold. +Reading out the innermost condition alone would claim the property is hidden far +more often than it is, so every term is shown and every term must hold before a +binding is pruned. A rule with one indeterminable term prunes nothing: the +binding is listed and you decide. + **`LIST WIDGETS` does not exist**, deliberately. `SHOW WIDGETS` already means widget *instances placed on pages*, and the definitions are `SELECT * FROM CATALOG.WIDGET_DEFINITIONS`. diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index fc985e3401..5eae3f583f 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -798,7 +798,13 @@ func ApplyVisibilityRules(object bson.D, propertyTypeIDs map[string]pages.Proper hidden[rule.PropertyKey] = false } // Several rules may govern one property; any one of them hiding it wins. - if rule.HiddenWhen.Hidden(values) { + // Within a rule, every term of its conjunction must hold — reading only + // HiddenWhen would hide a template in configurations the editor shows. + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + v, ok := values[c.PropertyKey] + return v, ok + }) + if determinable && fires { hidden[rule.PropertyKey] = true } } diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go index 8fa9e3f80a..44f0965c67 100644 --- a/mdl/executor/editorconfig_extract.go +++ b/mdl/executor/editorconfig_extract.go @@ -29,6 +29,11 @@ type editorConfigExtractStats struct { // the group properties that make an authored widget fail CE0463 (upstream #931). var hideCallRE = regexp.MustCompile(`hide(?:Property|Properties|NestedProperties)In\(`) +// hideCallTailRE matches a hide call's NAME where it ends a string, so +// stripGroupSiblings can tell a hide sibling from any other comma-separated +// expression. +var hideCallTailRE = regexp.MustCompile(`hide(?:Property|Properties|NestedProperties)In$`) + // aliasAssignRE finds `IDENT=OBJ.PROP` (a `var x=e.selection`-style alias). // Resolution is scoped to the enclosing function body (see enclosingAliases), // because minified editorConfig reuses single-letter identifiers across scopes. @@ -153,7 +158,42 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit if listKey != "" { itemIdent = enclosingForEachParam(js, callStart) } - cond, guardText, ok := parseGuard(js, callStart, itemIdent) + cond, guardText, ok, conjunctive := parseGuard(js, callStart, itemIdent) + // A guard inside `outer ? ( … inner && hide(x) … )` states only the INNER + // term; the branch runs on outer too. Collect the enclosing group guards + // so the rule carries the whole conjunction — Combo box nests three deep, + // and a rule keeping only the innermost term claims hidden in + // configurations the editor shows. + var extra []types.WidgetVisibilityCondition + if ok && conjunctive { + enclosing, enclosed := enclosingGroupConditions(js, callStart, itemIdent) + switch { + case enclosed: + // The whole chain read: the rule carries every term and is exact. + extra = dedupeConditions(cond, enclosing) + case impliesGroupGuard(js, callStart, cond): + // The group's condition is about the SAME property and holds + // wherever this one does, so the conjunction reduces to this + // condition alone. Maps: `"googleMaps"!==B.mapProvider ? + // (…, "openStreet"===B.mapProvider && hide([apiKey, apiKeyExp]))`. + default: + // The chain could not be read in full, so the terms cannot be + // stored. Keep the rule only for the keys the ternary's other + // branch hides anyway — there the single condition can + // under-report but never claim hidden where the editor shows it. + kept := keys[:0:0] + for _, k := range keys { + if hiddenInComplementaryBranch(js, callStart, k) { + kept = append(kept, k) + } + } + if len(kept) == 0 && len(condKeys) == 0 { + stats.SkippedComplex++ + continue + } + keys = kept + } + } if !ok { // ` ? && hide(...)` — a ternary THEN branch carrying a // second condition. Held back and resolved after the loop, once the @@ -193,7 +233,7 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit stats.SkippedComplex++ continue } - sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + condsSig(extra) if seen[sig] { continue } @@ -203,12 +243,13 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit PropertyKey: tk.key, ListPropertyKey: listKey, HiddenWhen: &cc, + And: extra, }) } } stats.Recognized++ for _, key := range keys { - sig := listKey + "\x00" + key + "\x00" + cond.PropertyKey + cond.Operator + cond.Value + cond.Scope + sig := listKey + "\x00" + key + "\x00" + cond.PropertyKey + cond.Operator + cond.Value + cond.Scope + condsSig(extra) if seen[sig] { continue } @@ -218,6 +259,7 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit PropertyKey: key, ListPropertyKey: listKey, HiddenWhen: &c, + And: extra, }) } } @@ -476,7 +518,11 @@ var nsPrefixRE = regexp.MustCompile(`[A-Za-z_$][\w$]*\.$`) // parseGuard reads the guard expression immediately preceding a hide call and // converts it to a WidgetVisibilityCondition. callStart points at the hide // function name; the connector just before it is `&&`, `||`, or `?`. -func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibilityCondition, string, bool) { +// The fourth result marks a guard that is one conjunct of a larger condition +// (it sits inside a grouping paren carrying its own guard). Such a rule is only +// safe when the ternary's other branch hides the same property anyway — see +// hiddenInComplementaryBranch, which the caller consults. +func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibilityCondition, string, bool, bool) { pre := strings.TrimRight(js[:callStart], " ") // Strip the widget-editor namespace prefix (any `.`, not just `_.`). if loc := nsPrefixRE.FindStringIndex(pre); loc != nil { @@ -485,6 +531,15 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil pre = strings.TrimRight(pre, " ") // Strip an optional grouping paren: `cond && ( hide(...), … )` groups several // hides under one condition; the first hide sits right after the `(`. + // + // The LATER hides in that group sit after a comma instead, so their guard is + // their sibling's. Skipping back over the preceding call expressions reaches + // the same `(` and the same condition. Without this the second hide in + // `cond ? (hide(a), hide(b))` gets no rule at all: `pre` ends with `,`, which + // is not a connector, and the property reads as always visible. Combobox's + // `attributeEnumeration` is that case — it is the FIRST hide of one group and + // the second of another, so it was extracted once and missed once. + pre = stripGroupSiblings(pre) if strings.HasSuffix(pre, "(") { pre = strings.TrimRight(pre[:len(pre)-1], " ") } @@ -514,17 +569,17 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil // with showLabel false. See the CE0463 that produced (ledger #104). cond, ok := ternaryCondition(pre[:len(pre)-1]) if !ok { - return types.WidgetVisibilityCondition{}, pre, false + return types.WidgetVisibilityCondition{}, pre, false, false } pre = cond falsy = true default: - return types.WidgetVisibilityCondition{}, pre, false + return types.WidgetVisibilityCondition{}, pre, false, false } guard, boundary := lastGuardExpr(pre) guard = stripReturnPrefix(guard) // getProperties' first statement is `return && hide…` if guard == "" { - return types.WidgetVisibilityCondition{}, guard, false + return types.WidgetVisibilityCondition{}, guard, false, false } // Skip guards nested inside a larger expression. A clean statement-level guard // is bounded by a statement separator (`,`, `;`, `{`, or start-of-input); a @@ -545,6 +600,33 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil // // The `&&` connector gets no such rule: there, hiding needs BOTH operands // truthy, which a single condition cannot express. + // A `,` boundary reads as statement-level but is not one when the guard sits + // inside a grouping paren that carries its OWN condition: the hide fires on + // the CONJUNCTION, and a single condition can only express one conjunct — + // which over-fires, hiding a property in configurations where the editor + // shows it. Datagrid is the case: + // + // e.pagination ? hide("showNumberOfRows") + // : (hide("showPagingButtons"), !1===e.showNumberOfRows && hide("pagingPosition")) + // + // pagingPosition is hidden only when pagination is off AND showNumberOfRows + // is false. Reading the `!1===` alone hides it whenever the row count is off, + // pagination or not — and `pagination` defaults ON, so that is the common + // configuration. Emitting no rule leaves it visible, which is the safe way to + // be wrong. + // Conjunctive only when the enclosing group's own condition is about the SAME + // object this guard reads — i.e. another of the widget's properties. + // + // editorConfig's getProperties also receives the target PLATFORM, and widgets + // branch on it: TreeNode hides its icon properties under + // `"web"===platform ? (e.advancedMode || hide([...])) : …`. That outer + // conjunct is not part of the widget's configuration and is always true for + // the pages MDL writes, so folding it away loses nothing — whereas dropping + // Datagrid's `e.pagination` conjunct, a real property with a real default, + // changes what the rule claims. + conjunctive := boundary == ',' && + insideOpenGroup(pre[:len(pre)-len(guard)]) && + sameReceiver(groupGuard(js, callStart), guard) switch boundary { case 0, ',', ';', '{': // clean @@ -553,17 +635,17 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil // (`outer ? inner && hide(...)`). Hand back the WHOLE expression, `?` // included, so the caller can split it and decide whether the else branch // makes the pair expressible — see ternaryThenCandidate (#238). - return types.WidgetVisibilityCondition{}, pre, false + return types.WidgetVisibilityCondition{}, pre, false, false case '&': if !falsy { - return types.WidgetVisibilityCondition{}, guard, false + return types.WidgetVisibilityCondition{}, guard, false, false } default: - return types.WidgetVisibilityCondition{}, guard, false + return types.WidgetVisibilityCondition{}, guard, false, false } aliases := enclosingAliases(js, callStart) c, ok := guardToCondition(guard, falsy, aliases, itemIdent) - return c, guard, ok + return c, guard, ok, conjunctive } // ternaryCondition returns the text preceding the `?` that matches a trailing @@ -689,6 +771,26 @@ var ( eqCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)==="([^"]*)"$`) neCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)!=="([^"]*)"$`) refRE = regexp.MustCompile(`^(!?)([A-Za-z_$][\w$.]*)$`) + + // Shapes that mean "the author has not picked anything", which editorConfig + // writes two ways. Both are narrower than falsy — see the `empty` operator's + // note in mdl/types/widget_visibility.go. + // null === ref / ref === null + nullCmpRE = regexp.MustCompile(`^null(===|!==)([A-Za-z_$][\w$.]*)$`) + nullCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)(===|!==)null$`) + // 0 === ref.length / ref.length === 0 + lenCmpRE = regexp.MustCompile(`^0(===|!==)([A-Za-z_$][\w$.]*)\.length$`) + lenCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)\.length(===|!==)0$`) + + // Minified booleans: terser writes `false` as `!1` and `true` as `!0`, so a + // guard reading `!1===e.showFooter` is `false===e.showFooter`. Mapped to + // eq/ne against the literal rather than to falsy/truthy, because `===false` + // does NOT fire on an unset property and falsy would. + boolCmpRE = regexp.MustCompile(`^!([01])(===|!==)([A-Za-z_$][\w$.]*)$`) + boolCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)(===|!==)!([01])$`) + + // ["a","b"].includes(ref) — a set-membership test over enum values. + includesRE = regexp.MustCompile(`^\[([^\]]*)\]\.includes\(([A-Za-z_$][\w$.]*)\)$`) ) // guardToCondition parses a single guard expression into a visibility @@ -735,6 +837,53 @@ func guardToCondition(guard string, falsy bool, aliases map[string]string, itemI } return types.WidgetVisibilityCondition{}, false } + // null === ref — "no datasource / action picked". Distinct from falsy. + if m := nullCmpRE.FindStringSubmatch(guard); m != nil { + return emptyCond(m[2], m[1], falsy, aliases, itemIdent) + } + if m := nullCmpRE2.FindStringSubmatch(guard); m != nil { + return emptyCond(m[1], m[2], falsy, aliases, itemIdent) + } + // 0 === ref.length — the same claim about a string or list property. + if m := lenCmpRE.FindStringSubmatch(guard); m != nil { + return emptyCond(m[2], m[1], falsy, aliases, itemIdent) + } + if m := lenCmpRE2.FindStringSubmatch(guard); m != nil { + return emptyCond(m[1], m[2], falsy, aliases, itemIdent) + } + // !1 === ref / ref === !0 — minified boolean comparison. + if m := boolCmpRE.FindStringSubmatch(guard); m != nil { + return boolCond(m[3], m[2], m[1], falsy, aliases, itemIdent) + } + if m := boolCmpRE2.FindStringSubmatch(guard); m != nil { + return boolCond(m[1], m[2], m[3], falsy, aliases, itemIdent) + } + // ["a","b"].includes(ref) — set membership. + if m := includesRE.FindStringSubmatch(guard); m != nil { + members := stringLitRE.FindAllStringSubmatch(m[1], -1) + if len(members) == 0 { + return types.WidgetVisibilityCondition{}, false + } + vals := make([]string, 0, len(members)) + for _, mm := range members { + if strings.ContainsRune(mm[1], ',') { + // A comma inside a member would make the joined Value ambiguous. + // Not observed in any marketplace widget; refuse rather than + // store a set that decodes wrong. + return types.WidgetVisibilityCondition{}, false + } + vals = append(vals, mm[1]) + } + key, scope, ok := resolveRef(m[2], aliases, itemIdent) + if !ok { + return types.WidgetVisibilityCondition{}, false + } + op := "in" + if falsy { + op = "notin" + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op, Value: strings.Join(vals, ","), Scope: scope}, true + } // bare ref (truthy) or !ref (falsy), combined with the connector polarity: // ref && hide → hide when ref truthy // ref || hide → hide when ref falsy (falsy==true here) @@ -756,6 +905,512 @@ func guardToCondition(guard string, falsy bool, aliases map[string]string, itemI return types.WidgetVisibilityCondition{}, false } +// impliesGroupGuard reports whether cond alone entails the enclosing group's +// condition, which happens when both constrain the same property and cond pins +// it to a value the group's condition accepts. The conjunction is then +// redundant and the single condition is exact rather than an over-fire. +func impliesGroupGuard(js string, callStart int, cond types.WidgetVisibilityCondition) bool { + if cond.Operator != "eq" || cond.PropertyKey == "" { + return false + } + text := groupGuard(js, callStart) + if text == "" { + return false + } + outer, ok := guardToCondition(text, groupIsElseBranch(js, callStart), enclosingAliases(js, callStart), "") + if !ok || outer.PropertyKey != cond.PropertyKey { + return false + } + // The group's condition must HOLD where this one does. Note the sense: the + // group guard is the condition under which the branch RUNS, and the branch + // running is what makes the hide fire, so it must be satisfied — Hidden() + // here is just the evaluator, not a claim about hiding. + return outer.Hidden(map[string]string{cond.PropertyKey: cond.Value}) +} + +// groupIsElseBranch reports whether the group enclosing callStart is the ELSE +// branch of a ternary, in which case its condition is negated. +func groupIsElseBranch(js string, callStart int) bool { + open := enclosingGroupOpen(js, callStart) + if open < 0 { + return false + } + head := strings.TrimRight(js[:open], " ") + return strings.HasSuffix(head, ":") || strings.HasSuffix(head, "||") +} + +// groupGuard returns the condition text attached to the grouping paren that +// encloses the hide at callStart ("" when there is none). +func groupGuard(js string, callStart int) string { + open := enclosingGroupOpen(js, callStart) + if open < 0 { + return "" + } + head := strings.TrimRight(js[:open], " ") + for _, c := range []string{"&&", "||"} { + if strings.HasSuffix(head, c) { + return stripReturnPrefix(trailingExpr(head[:len(head)-2])) + } + } + for _, c := range []string{"?", ":"} { + if strings.HasSuffix(head, c) { + head = head[:len(head)-1] + if c == ":" { + q := matchingTernaryQuestion(head) + if q < 0 { + return "" + } + head = head[:q] + } + // trailingExpr, not lastGuardExpr: the ternary is rarely the first + // thing in its function — ProgressCircle's is preceded by a whole + // `switch` — and taking everything back to the enclosing `{` hands + // the guard parser a fragment with an unbalanced `}`, which reads as + // "unsupported shape" and drops a sound rule. + return stripReturnPrefix(trailingExpr(head)) + } + } + return "" +} + +// matchingTernaryQuestion returns the index of the `?` matching the `:` that +// ends head, skipping over parenthesised groups and nested ternaries. A plain +// LastIndexByte finds the innermost `?` instead — in Maps that is a nested +// `B.geodecodeApiKey?…`, whose receiver is the widget, so the outer +// platform test read as a configuration conjunct and dropped a sound rule. +func matchingTernaryQuestion(head string) int { + depth, pending := 0, 0 + inStr := byte(0) + for i := len(head) - 1; i >= 0; i-- { + c := head[i] + if inStr != 0 { + if c == inStr && (i == 0 || head[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case ')': + depth++ + case '(': + if depth == 0 { + return -1 // ran out of the enclosing group + } + depth-- + case ':': + if depth == 0 { + pending++ + } + case '?': + if depth == 0 { + if pending == 0 { + return i + } + pending-- + } + case '{', ';': + if depth == 0 { + return -1 + } + } + } + return -1 +} + +// sameReceiver reports whether two guards read properties off the same object. +// An empty receiver on either side (a bare identifier, a literal-only guard) +// answers false: not demonstrably the same object, so not treated as a +// configuration conjunct. +func sameReceiver(a, b string) bool { + ra, rb := guardReceiver(a), guardReceiver(b) + return ra != "" && ra == rb +} + +// guardReceiver returns the object part of the first `obj.prop` reference in a +// guard, or "" when it reads no property off an object. +func guardReceiver(guard string) string { + m := receiverRE.FindStringSubmatch(guard) + if m == nil { + return "" + } + return m[1] +} + +var receiverRE = regexp.MustCompile(`([A-Za-z_$][\w$]*)\.[A-Za-z_$][\w$]*`) + +// dedupeConditions drops terms equal to the rule's own condition or to an +// earlier term. The innermost enclosing group is often the very group whose +// guard the hide already carries — `cond ? (hide(x), …)` gives the first hide +// its condition through the paren strip — and a conjunction repeating a term +// says nothing extra while reading as though it did. +func dedupeConditions(own types.WidgetVisibilityCondition, cs []types.WidgetVisibilityCondition) []types.WidgetVisibilityCondition { + seen := map[types.WidgetVisibilityCondition]bool{own: true} + out := cs[:0:0] + for _, c := range cs { + if seen[c] { + continue + } + seen[c] = true + out = append(out, c) + } + return out +} + +// condsSig renders conditions into a dedupe key. +func condsSig(cs []types.WidgetVisibilityCondition) string { + var b strings.Builder + for _, c := range cs { + b.WriteString("\x00") + b.WriteString(c.PropertyKey) + b.WriteString(c.Operator) + b.WriteString(c.Value) + b.WriteString(c.Scope) + } + return b.String() +} + +// enclosingGroupConditions walks outward from a hide call, collecting the +// condition of every grouping paren that encloses it, innermost first. +// +// ok is false when any link cannot be read as a condition — a platform test, a +// computed expression, a shape the guard vocabulary does not cover. The caller +// then emits nothing rather than a partial conjunction. +// +// A guard about something other than the widget's own properties is SKIPPED +// rather than failing the walk: editorConfig's getProperties also receives the +// target platform, and widgets branch on it (`"web"===platform ? (…)`). That +// term is not part of the configuration an MDL author writes and is always true +// for the pages MDL produces, so folding it away loses nothing. +func enclosingGroupConditions(js string, callStart int, itemIdent string) ([]types.WidgetVisibilityCondition, bool) { + var out []types.WidgetVisibilityCondition + at := callStart + for depth := 0; depth < maxGroupNesting; depth++ { + open := enclosingGroupOpen(js, at) + if open < 0 { + return out, true // reached statement level: the chain is complete + } + text := groupGuard(js, at) + if text == "" { + return nil, false + } + c, ok := guardToCondition(text, groupIsElseBranch(js, at), enclosingAliases(js, at), itemIdent) + if ok { + out = append(out, c) + } else if guardReceiver(text) != "" { + // Reads a property off an object but is not a shape we understand — + // a real term we cannot represent, so the conjunction is incomplete. + return nil, false + } + at = open + } + return nil, false +} + +// maxGroupNesting bounds the outward walk. Combo box reaches three; a file that +// nests further answers "cannot read" and its rules are simply not lifted. +const maxGroupNesting = 8 + +// hiddenInComplementaryBranch reports whether the OTHER branch of the ternary +// enclosing this hide also hides propertyKey. +// +// That is what separates a conjunctive guard that is merely imprecise from one +// that is wrong. Both of these hide on `outer AND inner`, and both are stored as +// `inner` alone: +// +// ProgressBar showLabel ? (… "text"!==labelType && hide("labelText")) +// : hide(["customLabel","labelText","labelType"]) +// Datagrid pagination ? hide("showNumberOfRows") +// : (…, !1===showNumberOfRows && hide("pagingPosition")) +// +// labelText is hidden in the else branch too, so `labelType != "text"` can only +// fail to fire where the widget hides it anyway — it over-lists, never hides a +// binding the author needs. pagingPosition is hidden in neither complementary +// case, so the same reading claims hidden with pagination on, which is its +// default. The first is kept, the second dropped. +// +// The scan is textual and bounded to the sibling branch. It answers "no" when +// the shape is not recognised, which drops the rule — the safe direction. +func hiddenInComplementaryBranch(js string, callStart int, propertyKey string) bool { + open := enclosingGroupOpen(js, callStart) + if open < 0 { + return false + } + // The group is one branch of `cond ? A : B`. Find the sibling branch: for a + // then-group `? ( … )` it follows the matching `)`; for an else-group + // `: ( … )` it is the text between the `?` and this `:`. + pre := strings.TrimRight(js[:open], " ") + var sibling string + switch { + case strings.HasSuffix(pre, "?"): + close := matchingCloseParen(js, open) + if close < 0 { + return false + } + rest := js[close+1:] + i := strings.IndexByte(rest, ':') + if i < 0 { + return false + } + sibling = rest[i+1:] + if len(sibling) > complementScanLimit { + sibling = sibling[:complementScanLimit] + } + case strings.HasSuffix(pre, ":"): + q := strings.LastIndexByte(pre[:len(pre)-1], '?') + if q < 0 { + return false + } + sibling = pre[q+1 : len(pre)-1] + default: + return false + } + return mentionsHiddenProperty(sibling, propertyKey) +} + +// complementScanLimit bounds the then-branch scan, whose end is not delimited by +// a paren the way the else-branch's is. Every observed ternary branch is far +// shorter; a longer one simply answers "no" and drops the rule. +const complementScanLimit = 4000 + +// mentionsHiddenProperty reports whether s contains a hide call naming +// propertyKey as a quoted argument. +func mentionsHiddenProperty(s, propertyKey string) bool { + lit := `"` + propertyKey + `"` + for _, loc := range hideCallRE.FindAllStringIndex(s, -1) { + args, ok := balancedArgs(s, loc[1]) + if !ok { + continue + } + if strings.Contains(args, lit) { + return true + } + } + return false +} + +// enclosingGroupOpen returns the index of the unmatched '(' that opens the group +// containing callStart, or -1 when the hide is not inside one. +func enclosingGroupOpen(js string, callStart int) int { + depth := 0 + inStr := byte(0) + for i := callStart - 1; i >= 0; i-- { + c := js[i] + if inStr != 0 { + if c == inStr && (i == 0 || js[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case ')': + depth++ + case '(': + if depth == 0 { + head := strings.TrimRight(js[:i], " ") + if strings.HasSuffix(head, "?") || strings.HasSuffix(head, ":") || + strings.HasSuffix(head, "&&") || strings.HasSuffix(head, "||") { + return i + } + return -1 + } + depth-- + case '{', ';': + if depth == 0 { + return -1 + } + } + } + return -1 +} + +// matchingCloseParen returns the index of the ')' matching the '(' at open. +func matchingCloseParen(js string, open int) int { + depth := 0 + inStr := byte(0) + for i := open; i < len(js); i++ { + c := js[i] + if inStr != 0 { + if c == inStr && js[i-1] != '\\' { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case '(': + depth++ + case ')': + depth-- + if depth == 0 { + return i + } + } + } + return -1 +} + +// insideOpenGroup reports whether prefix leaves a grouping paren open — i.e. the +// guard that follows it is one operand inside `cond ? ( … )` rather than a +// statement of its own. +// +// The scan is backwards and stops at the enclosing statement (`{` or `;` at +// depth zero), so the parens of an enclosing `function(a,b){…}` or a +// `.forEach((function(o,r){…}))` are not miscounted as an open group — without +// that bound every nested (object-list) rule would read as conjunctive. +func insideOpenGroup(prefix string) bool { + depth := 0 + inStr := byte(0) + for i := len(prefix) - 1; i >= 0; i-- { + c := prefix[i] + if inStr != 0 { + if c == inStr && (i == 0 || prefix[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case ')': + depth++ + case '(': + if depth == 0 { + // An unmatched open paren — but only a paren that FOLLOWS a + // conditional connector groups a guarded branch. `switch(a,b,c)`, + // `if(...)` and an ordinary call also leave one open, and their + // contents are not conditioned on anything, so treating them as + // groups would drop sound rules: Timeline writes its hides inside + // `switch(A, B, e.groupByKey)`. + head := strings.TrimRight(prefix[:i], " ") + return strings.HasSuffix(head, "?") || strings.HasSuffix(head, ":") || + strings.HasSuffix(head, "&&") || strings.HasSuffix(head, "||") + } + depth-- + case '{', ';': + if depth == 0 { + return false // reached the enclosing statement cleanly + } + } + } + return false +} + +// stripGroupSiblings walks back over `hide…(…),` siblings so a hide that is not +// the first in a comma group is attributed to the group's guard. +// +// Only a preceding *hide call* is skipped, never an arbitrary expression: a +// comma in editorConfig also separates object literals and array elements, and +// skipping one of those would attach a guard belonging to something else. The +// walk stops at anything it does not recognise, which leaves pre where it was +// and the hide unattributed — today's behaviour, and the safe direction. +func stripGroupSiblings(pre string) string { + walked := stripGroupSiblingsRaw(pre) + // Commit only when the walk lands on the group's own `(`. Landing anywhere + // else means the "sibling" was not a comma-group member at all: in + // `A ? B : hide(x), hide(y)` the text before hide(y) is a ternary ELSE + // branch, and attributing y to `A` is simply wrong — Maps hides `advanced` + // there, and the mis-read made it "hidden when geodecodeApiKey is not set". + if strings.HasSuffix(walked, "(") { + return walked + } + return pre +} + +func stripGroupSiblingsRaw(pre string) string { + for { + trimmed := strings.TrimRight(pre, " ") + if !strings.HasSuffix(trimmed, ",") { + return pre + } + body := strings.TrimRight(trimmed[:len(trimmed)-1], " ") + if !strings.HasSuffix(body, ")") { + return pre + } + open := matchingOpenParen(body) + if open < 0 { + return pre + } + head := strings.TrimRight(body[:open], " ") + loc := hideCallTailRE.FindStringIndex(head) + if loc == nil || loc[1] != len(head) { + return pre // not a hide call — do not cross this comma + } + pre = strings.TrimRight(head[:loc[0]], " ") + if l := nsPrefixRE.FindStringIndex(pre); l != nil { + pre = strings.TrimRight(pre[:l[0]], " ") + } + } +} + +// matchingOpenParen returns the index of the '(' matching the ')' that ends s, +// or -1 when it is unbalanced. Quoted strings are skipped so a paren inside a +// caption literal does not throw the count off. +func matchingOpenParen(s string) int { + depth := 0 + inStr := byte(0) + for i := len(s) - 1; i >= 0; i-- { + c := s[i] + if inStr != 0 { + if c == inStr && (i == 0 || s[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case ')': + depth++ + case '(': + depth-- + if depth == 0 { + return i + } + } + } + return -1 +} + +// emptyCond builds an empty/notempty condition for `null===ref` and +// `0===ref.length`. cmp is the JS operator as written; falsy is the connector +// polarity, and the two compose — `!==` under `||` cancels back to "empty". +func emptyCond(ref, cmp string, falsy bool, aliases map[string]string, itemIdent string) (types.WidgetVisibilityCondition, bool) { + key, scope, ok := resolveRef(ref, aliases, itemIdent) + if !ok { + return types.WidgetVisibilityCondition{}, false + } + wantEmpty := (cmp == "===") != falsy // XOR + op := "notempty" + if wantEmpty { + op = "empty" + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op, Scope: scope}, true +} + +// boolCond builds an eq/ne condition against a minified boolean literal, where +// digit is terser's "1" for false (`!1`) and "0" for true (`!0`). +func boolCond(ref, cmp, digit string, falsy bool, aliases map[string]string, itemIdent string) (types.WidgetVisibilityCondition, bool) { + key, scope, ok := resolveRef(ref, aliases, itemIdent) + if !ok { + return types.WidgetVisibilityCondition{}, false + } + lit := "false" + if digit == "0" { + lit = "true" + } + wantEq := (cmp == "===") != falsy // XOR + op := "ne" + if wantEq { + op = "eq" + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op, Value: lit, Scope: scope}, true +} + // resolveRef turns a guard reference into a property key and the scope that key // belongs to: `obj.prop` yields `prop`; a bare identifier is looked up in the // scope alias map. A bare identifier with no alias (e.g. a computed local) is diff --git a/mdl/executor/editorconfig_shapes_test.go b/mdl/executor/editorconfig_shapes_test.go new file mode 100644 index 0000000000..e72cea5b32 --- /dev/null +++ b/mdl/executor/editorconfig_shapes_test.go @@ -0,0 +1,243 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// The guard vocabulary editorConfig actually uses. Each shape below was found +// unrecognised in a real marketplace widget, and each is stated with the +// polarity flip that a `||` connector or a ternary ELSE branch applies — a +// shape that lifts under `&&` and inverts wrongly under `||` produces a rule +// that hides a property in exactly the configuration the editor shows it. +func TestGuardShapes(t *testing.T) { + tests := []struct { + name string + guard string + falsy bool + key string + op string + value string + }{ + // `null === x` is "the author picked no datasource / no action". It is + // NOT falsy: "false" and "0" are values a real property can hold. + {"null on the left", `null===t.optionsSourceAssociationDataSource`, false, "optionsSourceAssociationDataSource", "empty", ""}, + {"null on the right", `t.someDataSource===null`, false, "someDataSource", "empty", ""}, + {"null inverted", `null===t.someDataSource`, true, "someDataSource", "notempty", ""}, + {"not-null", `null!==t.someDataSource`, false, "someDataSource", "notempty", ""}, + + // terser writes `false` as `!1` and `true` as `!0`. Mapped to eq/ne + // against the literal, not to falsy/truthy: `===false` does not fire on + // an unset property, and falsy would. + {"minified false", `!1===t.showFooter`, false, "showFooter", "eq", "false"}, + {"minified true", `!0===t.showFooter`, false, "showFooter", "eq", "true"}, + {"minified false inverted", `!1===t.showFooter`, true, "showFooter", "ne", "false"}, + {"minified false on the right", `t.showFooter===!1`, false, "showFooter", "eq", "false"}, + + // `0 === x.length` is the same "nothing picked" claim about a list or + // string. The property is x, not x.length. + {"empty length", `0===t.databaseAttributeString.length`, false, "databaseAttributeString", "empty", ""}, + {"empty length inverted", `0===t.databaseAttributeString.length`, true, "databaseAttributeString", "notempty", ""}, + {"non-empty length", `t.databaseAttributeString.length!==0`, false, "databaseAttributeString", "notempty", ""}, + + // set membership over enum values + {"includes", `["enumeration","boolean"].includes(t.optionsSourceType)`, false, "optionsSourceType", "in", "enumeration,boolean"}, + {"includes inverted", `["enumeration","boolean"].includes(t.optionsSourceType)`, true, "optionsSourceType", "notin", "enumeration,boolean"}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + c, ok := guardToCondition(tc.guard, tc.falsy, map[string]string{}, "") + if !ok { + t.Fatalf("guard not recognised: %s", tc.guard) + } + if c.PropertyKey != tc.key || c.Operator != tc.op || c.Value != tc.value { + t.Errorf("got %s %s %q, want %s %s %q", + c.PropertyKey, c.Operator, c.Value, tc.key, tc.op, tc.value) + } + }) + } +} + +// A shape outside the vocabulary must yield NO rule rather than a guessed one: +// a wrong rule hides a property the author is using, which is the failure this +// whole area keeps producing. This is the control for the table above — without +// it, a `guardToCondition` that returned a rule for everything would pass. +func TestGuardShapesRefuseWhatTheyCannotRead(t *testing.T) { + for _, guard := range []string{ + `t.items.filter(function(i){return i.on}).length>2`, // computed + `3===t.count`, // a number we do not model + `null===someLocal`, // unresolvable bare identifier + `["a"].includes(t.x)&&t.y`, // compound + `0===t.a.length&&0===t.b.length`, // compound + } { + if c, ok := guardToCondition(guard, false, map[string]string{}, ""); ok { + t.Errorf("guard %q produced a rule (%s %s %q); it should be refused", + guard, c.PropertyKey, c.Operator, c.Value) + } + } +} + +// A hide that is not the FIRST in a comma group is guarded by its sibling's +// condition: `cond ? (hide(a), hide(b))` hides b under cond just as much as a. +// Attributing only the first left the rest reading as always visible. +func TestCommaGroupSiblingsShareTheGuard(t *testing.T) { + js := `function getProperties(e,r,t){` + + `return"text"===e.headerType?(M.hidePropertyIn(r,e,"headerContent"),M.hidePropertyIn(r,e,"openNodeOn")):0,r}` + + rules, _ := extractVisibilityRulesFromJS(js) + got := map[string]string{} + for _, r := range rules { + if r.HiddenWhen != nil { + got[r.PropertyKey] = r.HiddenWhen.PropertyKey + " " + r.HiddenWhen.Operator + " " + r.HiddenWhen.Value + } + } + for _, key := range []string{"headerContent", "openNodeOn"} { + if got[key] != `headerType eq text` { + t.Errorf("%s: got %q, want %q", key, got[key], "headerType eq text") + } + } +} + +// The walk must not cross a comma that is not a group separator. In +// `A ? B : hide(x), hide(y)` the text before hide(y) is a ternary ELSE branch, +// and attributing y to A is simply wrong — Maps hides `advanced` in exactly +// that position, and the mis-read made it "hidden when geodecodeApiKey is not +// set", a condition with nothing to do with it. +func TestCommaWalkStopsOutsideAGroup(t *testing.T) { + js := `function getProperties(B,F,A){` + + `return B.geodecodeApiKey?u.hidePropertyIn(F,B,"geodecodeApiKeyExp"):u.hidePropertyIn(F,B,"geodecodeApiKey"),` + + `u.hidePropertyIn(F,B,"advanced"),F}` + + rules, _ := extractVisibilityRulesFromJS(js) + for _, r := range rules { + if r.PropertyKey == "advanced" { + t.Errorf("attributed `advanced` to a guard it does not have: %s %s %q", + r.HiddenWhen.PropertyKey, r.HiddenWhen.Operator, r.HiddenWhen.Value) + } + } + // Control: the two properties that DO have that guard still get it, so the + // test is not passing because extraction stopped altogether. + var seen int + for _, r := range rules { + if r.PropertyKey == "geodecodeApiKey" || r.PropertyKey == "geodecodeApiKeyExp" { + seen++ + } + } + if seen != 2 { + t.Errorf("expected the two guarded rules to survive, got %d", seen) + } +} + +// A guard inside `outer ? ( … inner && hide(x) … )` states only the INNER term. +// Storing that alone claims hidden wherever inner holds, outer or not — which +// is wrong whenever outer is false. The rule must carry both terms. +func TestConjunctionCarriesEveryTerm(t *testing.T) { + js := `function getProperties(t,e){` + + `return"context"===e.source?(x.hidePropertiesIn(t,e,["a"]),!1===e.showFooter&&x.hidePropertiesIn(t,e,["menuFooterContent"])):0,t}` + + rules, _ := extractVisibilityRulesFromJS(js) + var got *types.WidgetVisibilityRule + for i := range rules { + if rules[i].PropertyKey == "menuFooterContent" { + got = &rules[i] + } + } + if got == nil { + t.Fatal("no rule for menuFooterContent") + } + conds := got.Conditions() + if len(conds) != 2 { + t.Fatalf("want 2 conditions, got %d: %+v", len(conds), conds) + } + // It must not fire on the inner term alone. + fires, determinable := got.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + return map[string]string{"showFooter": "false", "source": "database"}[c.PropertyKey], true + }) + if !determinable || fires { + t.Error("fired with source=database: the outer term was dropped") + } + // And it must fire when both hold — the control, without which the test + // would pass against a rule that never fires at all. + fires, determinable = got.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + return map[string]string{"showFooter": "false", "source": "context"}[c.PropertyKey], true + }) + if !determinable || !fires { + t.Error("did not fire with both terms satisfied") + } +} + +// A conjunction is only as decidable as its least decidable term. One unknown +// value makes the whole rule indeterminable, so callers keep asking for the +// binding rather than guessing. +func TestConjunctionIsIndeterminableIfAnyTermIs(t *testing.T) { + c1 := types.WidgetVisibilityCondition{PropertyKey: "a", Operator: "eq", Value: "1"} + c2 := types.WidgetVisibilityCondition{PropertyKey: "b", Operator: "eq", Value: "2"} + r := types.WidgetVisibilityRule{PropertyKey: "x", HiddenWhen: &c1, And: []types.WidgetVisibilityCondition{c2}} + + _, determinable := r.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + if c.PropertyKey == "a" { + return "1", true + } + return "", false // b unknown + }) + if determinable { + t.Error("reported determinable with an unknown term") + } + // Control: known values make it decidable and firing. + fires, determinable := r.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + return map[string]string{"a": "1", "b": "2"}[c.PropertyKey], true + }) + if !determinable || !fires { + t.Error("did not fire when every term is known and satisfied") + } +} + +// The platform argument is not part of the widget's configuration: TreeNode +// hides its icon properties under `"web"===platform ? (e.advancedMode || …)`, +// and MDL only writes web pages. Folding that term away keeps the rule; keeping +// it would make the rule indeterminable and over-list three bindings. +func TestPlatformTermIsNotAConfigurationConjunct(t *testing.T) { + js := `function getProperties(e,r,t){` + + `return"web"===t?(M.transformGroupsIntoTabs(r),e.advancedMode||M.hidePropertiesIn(r,e,["showIcon","animate"])):M.hidePropertyIn(r,e,"advancedMode"),r}` + + rules, _ := extractVisibilityRulesFromJS(js) + for _, want := range []string{"showIcon", "animate"} { + var found bool + for _, r := range rules { + if r.PropertyKey == want && r.HiddenWhen != nil && + r.HiddenWhen.PropertyKey == "advancedMode" && r.HiddenWhen.Operator == "falsy" { + found = true + if len(r.And) != 0 { + t.Errorf("%s carries a platform term: %+v", want, r.And) + } + } + } + if !found { + t.Errorf("lost the rule for %s", want) + } + } +} + +// The English rendering must show every term. Reading out only the innermost +// makes a narrow rule look like a broad one, which is how a reader concludes a +// property is hidden far more often than it is. +func TestRuleTextJoinsEveryTerm(t *testing.T) { + r := types.WidgetVisibilityRule{ + PropertyKey: "x", + HiddenWhen: &types.WidgetVisibilityCondition{PropertyKey: "showFooter", Operator: "eq", Value: "false"}, + And: []types.WidgetVisibilityCondition{ + {PropertyKey: "source", Operator: "eq", Value: "context"}, + }, + } + got := ruleText(r) + for _, want := range []string{`showFooter = "false"`, " and ", `source = "context"`} { + if !strings.Contains(got, want) { + t.Errorf("rendering %q is missing %q", got, want) + } + } +} diff --git a/mdl/executor/validate_widget_hidden.go b/mdl/executor/validate_widget_hidden.go index 5d86fbf328..2e2def7f7f 100644 --- a/mdl/executor/validate_widget_hidden.go +++ b/mdl/executor/validate_widget_hidden.go @@ -118,15 +118,17 @@ func validateWidgetItemVisibility(parent *ast.WidgetV3, item *ast.WidgetV3, if !itemExplicit[strings.ToLower(rule.PropertyKey)] { continue // the author did not set this sub-property } - values := widgetValues - if rule.HiddenWhen.Scope == types.ConditionScopeItem { - values = itemValues - } - condVal, known := values[strings.ToLower(rule.HiddenWhen.PropertyKey)] - if !known { - continue // condition value indeterminable — don't guess - } - if !rule.HiddenWhen.Hidden(map[string]string{rule.HiddenWhen.PropertyKey: condVal}) { + // Each term is resolved in its OWN scope: a nested rule mixes conditions + // about the list item with conditions about the widget. + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + values := widgetValues + if c.Scope == types.ConditionScopeItem { + values = itemValues + } + v, ok := values[strings.ToLower(c.PropertyKey)] + return v, ok + }) + if !determinable || !fires { continue } value := itemValues[strings.ToLower(rule.PropertyKey)] diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index e2b3f3a721..edbead1bab 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -317,12 +317,14 @@ func validateWidgetVisibility(w *ast.WidgetV3, registry *WidgetRegistry, locatio if !explicit[strings.ToLower(rule.PropertyKey)] { continue // user didn't set this property — nothing to warn about } - condVal, known := values[strings.ToLower(rule.HiddenWhen.PropertyKey)] - if !known { - continue // condition value indeterminable — don't guess - } - if !rule.HiddenWhen.Hidden(map[string]string{rule.HiddenWhen.PropertyKey: condVal}) { - continue + // EVERY term of the rule's conjunction must hold; a rule read through + // HiddenWhen alone over-fires (see WidgetVisibilityRule.And). + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + v, ok := values[strings.ToLower(c.PropertyKey)] + return v, ok + }) + if !determinable || !fires { + continue // indeterminable, or the configuration does not hide it } out = append(out, hiddenPropertyViolation(locationPrefix, w.Name, def.MDLName, "", rule, values[strings.ToLower(rule.PropertyKey)], diff --git a/mdl/executor/widget_describe.go b/mdl/executor/widget_describe.go index a61b14b839..06b672fd51 100644 --- a/mdl/executor/widget_describe.go +++ b/mdl/executor/widget_describe.go @@ -114,6 +114,9 @@ type DescribedRule struct { // required only where visible, and Combo box lists eleven bindings of which // its mutually exclusive options-source modes leave about two. Cond *types.WidgetVisibilityCondition `json:"-"` + // Rule is the whole rule, conjunction included. Cond above is only its FIRST + // term, kept for readers that predate conjunctions; evaluate Rule. + Rule types.WidgetVisibilityRule `json:"-"` // Nested marks a rule about an object-list ITEM's property rather than the // widget's own. Those are evaluated against the item, never the widget. Nested bool `json:"-"` @@ -350,8 +353,9 @@ func rulesToDescribed(rules []types.WidgetVisibilityRule) []DescribedRule { } out = append(out, DescribedRule{ Property: r.PropertyKey, - HiddenWhen: conditionText(r.HiddenWhen), + HiddenWhen: ruleText(r), Cond: r.HiddenWhen, + Rule: r, Nested: r.Nested(), }) } @@ -359,6 +363,28 @@ func rulesToDescribed(rules []types.WidgetVisibilityRule) []DescribedRule { return out } +// ruleText renders a rule's whole condition as readable English, joining a +// conjunction with "and" so the reader sees every term rather than the +// innermost one, which alone reads as a far broader claim than the rule makes. +func ruleText(r types.WidgetVisibilityRule) string { + conds := r.Conditions() + parts := make([]string, 0, len(conds)) + for i := range conds { + parts = append(parts, conditionText(&conds[i])) + } + return strings.Join(parts, " and ") +} + +// rule returns the described rule in structured form, falling back to the +// single Cond for a DescribedRule built without one (the JSON shape and older +// callers carry only the condition). +func (r DescribedRule) rule() types.WidgetVisibilityRule { + if r.Rule.HiddenWhen != nil { + return r.Rule + } + return types.WidgetVisibilityRule{PropertyKey: r.Property, HiddenWhen: r.Cond} +} + // conditionText renders a visibility condition as readable English. func conditionText(c *types.WidgetVisibilityCondition) string { switch c.Operator { @@ -370,6 +396,14 @@ func conditionText(c *types.WidgetVisibilityCondition) string { return fmt.Sprintf("%s is set", c.PropertyKey) case "falsy": return fmt.Sprintf("%s is not set", c.PropertyKey) + case "empty": + return fmt.Sprintf("%s is empty", c.PropertyKey) + case "notempty": + return fmt.Sprintf("%s is not empty", c.PropertyKey) + case "in": + return fmt.Sprintf("%s is one of %s", c.PropertyKey, strings.ReplaceAll(c.Value, ",", ", ")) + case "notin": + return fmt.Sprintf("%s is not one of %s", c.PropertyKey, strings.ReplaceAll(c.Value, ",", ", ")) default: return fmt.Sprintf("%s %s %q", c.PropertyKey, c.Operator, c.Value) } @@ -738,6 +772,33 @@ func pageBodyParses(body string) bool { return len(errs) == 0 } +// isEmptinessOperator reports whether an operator's whole question is "did the +// author pick anything", making an unrecorded value determinable rather than +// unknown. +func isEmptinessOperator(op string) bool { + return op == "empty" || op == "notempty" +} + +// declaresProperty reports whether the widget declares a property with this key, +// at any nesting depth. A condition naming something the widget does not declare +// is not a property that is merely unset — it is a rule we misread, and guessing +// "" for it would invent a verdict. +func declaresProperty(d WidgetDescription, key string) bool { + var walk func(props []DescribedProperty) bool + walk = func(props []DescribedProperty) bool { + for _, p := range props { + if strings.EqualFold(p.Key, key) { + return true + } + if walk(p.Children) { + return true + } + } + return false + } + return walk(d.Properties) +} + // exampleValues is the configuration the example describes: each scalar // property's default, which is also what the example writes for the required // ones. Visibility rules are evaluated against this. @@ -823,10 +884,27 @@ func hiddenUnder(d WidgetDescription, propertyKey string) bool { if r.Nested || r.Cond == nil || !strings.EqualFold(r.Property, propertyKey) { continue } - if _, known := values[r.Cond.PropertyKey]; !known { - continue // indeterminable — do not guess, keep asking for it - } - if r.Cond.Hidden(values) { + fires, determinable := r.rule().Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + if v, known := values[c.PropertyKey]; known { + return v, true + } + // `empty`/`notempty` ask whether the author picked anything, and for + // a property the widget DECLARES, "nothing recorded" is the answer, + // not a gap: an unbound datasource or action holds "". Without this + // the two operators could never fire, because exampleValues records + // a property only when it has a non-empty default — which is exactly + // what an unset datasource does not have. + // + // Deliberately scoped to these two operators. Feeding "" to eq/ne/ + // truthy/falsy would change verdicts for rules that already exist, + // in the direction of hiding a binding the author needs — the one + // failure this whole area keeps producing. + if !isEmptinessOperator(c.Operator) || !declaresProperty(d, c.PropertyKey) { + return "", false // indeterminable — do not guess, keep asking for it + } + return "", true + }) + if determinable && fires { return true } } diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index f0b4a1e54f..bd478d3e7f 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -697,13 +697,43 @@ func (e *PluggableWidgetEngine) hiddenUnnamedProperties(def *WidgetDefinition, w continue // still indeterminable — never guess } } - if rule.HiddenWhen.Hidden(map[string]string{rule.HiddenWhen.PropertyKey: condVal}) { + if firesUnder(rule, condVal, values, stored, defaults) { out[key] = defaults[defaultsKey("", rule.PropertyKey)] } } return out } +// firesUnder evaluates every term of a rule's conjunction, resolving each the +// same way the first term was resolved above: the widget's mapped values, then +// the stored template, then the declared default. firstVal is the already +// resolved value of the first term, so that work is not repeated. +// +// A term that stays indeterminable makes the whole rule not fire — the +// serializer never guesses, which is why an unresolvable condition leaves the +// template's value alone. +func firesUnder(rule types.WidgetVisibilityRule, firstVal string, values, stored, defaults map[string]string) bool { + first := true + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + if first { + first = false + return firstVal, true + } + if v, ok := values[c.PropertyKey]; ok && v != "" { + return v, true + } + if v, ok := stored[c.PropertyKey]; ok && v != "" { + return v, true + } + v, ok := defaults[defaultsKey("", c.PropertyKey)] + if !ok || v == "" { + return "", false + } + return v, true + }) + return determinable && fires +} + // visibilityRules is the widget's editorConfig visibility rules: from the // .def.json when it carries them, otherwise a live lift from the installed .mpk. // Both consumers must use the same list — reading the .def.json field directly diff --git a/mdl/types/widget_visibility.go b/mdl/types/widget_visibility.go index 7c3b054528..9d9ff3d51c 100644 --- a/mdl/types/widget_visibility.go +++ b/mdl/types/widget_visibility.go @@ -2,6 +2,8 @@ package types +import "strings" + // WidgetVisibilityRule declares that a pluggable widget property is hidden // under certain configurations of the same widget. Pluggable widgets express // this in their compiled editorConfig.js via Mendix's hidePropertyIn / @@ -22,6 +24,52 @@ type WidgetVisibilityRule struct { // its PropertyKey does not name a property of the widget itself. ListPropertyKey string `json:"listPropertyKey,omitempty"` HiddenWhen *WidgetVisibilityCondition `json:"hiddenWhen,omitempty"` + // And carries the REST of a conjunction whose first term is HiddenWhen. A + // widget's editorConfig nests its branches — Combo box reaches + // `source=="context" && optionsSourceType=="association" && showFooter==false` + // three levels deep — and a rule that keeps only the innermost term claims + // hidden in configurations the editor shows. Every term must hold. + // + // Consumers MUST evaluate these; Conditions() and Fires() exist so they do + // not have to remember. A reader that looks only at HiddenWhen silently + // over-fires, which is why the extractor previously refused to lift such a + // rule at all rather than store a partial one. + And []WidgetVisibilityCondition `json:"and,omitempty"` +} + +// Conditions returns every condition that must hold for the rule to fire. +func (r WidgetVisibilityRule) Conditions() []WidgetVisibilityCondition { + if r.HiddenWhen == nil { + return nil + } + out := make([]WidgetVisibilityCondition, 0, 1+len(r.And)) + out = append(out, *r.HiddenWhen) + return append(out, r.And...) +} + +// Fires reports whether the rule hides its property, given a lookup of the +// current value of a condition's property in a given scope. +// +// determinable is false when ANY condition's property has no known value — a +// conjunction is only as decidable as its least decidable term, and guessing +// one would resurrect exactly the over-firing this type exists to prevent. +// Callers treat indeterminable as "not hidden" and keep asking for the binding. +func (r WidgetVisibilityRule) Fires(lookup func(c WidgetVisibilityCondition) (string, bool)) (fires, determinable bool) { + conds := r.Conditions() + if len(conds) == 0 { + return false, false + } + all := true + for _, c := range conds { + v, ok := lookup(c) + if !ok { + return false, false + } + if !c.Hidden(map[string]string{c.PropertyKey: v}) { + all = false + } + } + return all, true } // Nested reports whether the rule targets an item sub-property of an object list @@ -32,10 +80,21 @@ func (r WidgetVisibilityRule) Nested() bool { return r.ListPropertyKey != "" } // widget's current property values. Operators cover the dominant patterns // observed in marketplace editorConfig.js files: // -// eq — the named property equals Value -// ne — the named property differs from Value -// truthy — the named property is set / non-empty / not "false"/"0" -// falsy — the named property is unset / empty / "false" / "0" +// eq — the named property equals Value +// ne — the named property differs from Value +// truthy — the named property is set / non-empty / not "false"/"0" +// falsy — the named property is unset / empty / "false" / "0" +// empty — the named property is the empty string +// notempty — the named property is anything but the empty string +// in — the named property is one of Value's comma-separated entries +// notin — the named property is none of them +// +// `empty` is NOT `falsy`: editorConfig writes `null===e.someDataSource` and +// `0===e.someList.length` for "the author has not picked one", which is a +// narrower claim than falsy — "false" and "0" are real values a boolean or a +// number property can hold, and treating them as unset hides a property the +// author is actively using. Keeping the two apart is what lets `empty` fire on +// a datasource without also firing on `false`. // // Conditions that don't fit (composite logic, runtime data lookups) are left // unset, which evaluates to "not hidden" so serialization falls back to the @@ -74,11 +133,35 @@ func (c *WidgetVisibilityCondition) Hidden(values map[string]string) bool { return isTruthyPrimitive(current) case "falsy": return !isTruthyPrimitive(current) + case "empty": + return current == "" + case "notempty": + return current != "" + case "in": + return containsCSV(c.Value, current) + case "notin": + return !containsCSV(c.Value, current) default: return false } } +// containsCSV reports whether want is one of csv's comma-separated entries. +// The set comes from an editorConfig `["a","b"].includes(e.prop)` guard, whose +// members are property keys and enum values — neither of which can contain a +// comma, so no escaping is needed. +func containsCSV(csv, want string) bool { + if csv == "" { + return false + } + for _, part := range strings.Split(csv, ",") { + if part == want { + return true + } + } + return false +} + // isTruthyPrimitive mirrors how Mendix treats a boolean/enum primitive in // editorConfig.js: empty, "false", and "0" are falsy; everything else truthy. func isTruthyPrimitive(v string) bool { From e6596c577352f40af8cd802361900c8943e63177 Mon Sep 17 00:00:00 2001 From: Ako Date: Wed, 9 Sep 2026 07:55:27 +0000 Subject: [PATCH 06/19] feat(navigation): author ON SYNC ERROR THROW|CONTINUE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Studio Pro's "Throw error when server rejects objects during synchronization", stored as the profile-level ThrowPartialSyncError. The last unauthorable field in the offline sync dialog. Spelled with the phrase MDL already uses for failure handling — a microflow's ON ERROR CONTINUE / ON ERROR ROLLBACK — so it needs no new token and reads as something already learned. The alternatives were worse for concrete reasons. camelCase (`on syncReject throw`) cannot work: MDL's lexer is case-insensitive across all 563 keyword tokens, so syncReject and syncreject lex identically and the casing would be a distinction the grammar cannot enforce; there is also no camelCase precedent, SIGN_OUT being the one compound keyword. And REJECT, the platform's own word, appears ~500 times across the examples and skills, because approve/reject is one of the commonest things a workflow models — not a word to claim as a keyword for one checkbox. Neither modelsdk/gen nor generated/metamodel declares the property (zero occurrences in each, measured on ako/TestApp), so there is no typed accessor: it is read from element.Base.Raw() and written as a raw key on both engines. Web profiles only — both of TestApp's carry it; whether a native profile does is unmeasured, so nativeNavProfileFromGen is left alone rather than given a default nothing has verified. The spec field is a POINTER. The property is a bare bool with no unset value of its own, so a non-pointer would reset the flag on every rewrite that never mentions the clause — silently, on a property nothing else reports. Absent reads as TRUE, matching every reference profile and Studio Pro's checked-by- default box, so a document lacking the key is not flipped to "do not throw". DESCRIBE emits the clause only when it is not the default, so navigation scripts do not all gain a line that says what would happen anyway. A control caught a hole in the test rather than in the code. The first fixture stored false, which is also the zero value, so a writer that ignored the spec entirely produced identical output and the assertion held either way — the control failed to fail. The test now exercises BOTH stored values, and with the pointer check removed reports "changed a stored true to false". Second time in this feature after CompatibilityMode, and recorded as a finding: when every real document agrees on a value, a fixture drawn from real documents cannot test preservation. Verified on ako/TestApp: the flag flips to false, the untouched Responsive profile stays true, a rewrite that never mentions the clause leaves false intact, and mx check reports 0 errors. Co-Authored-By: Claude Opus 5 --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .../skills/mendix/manage-navigation/SKILL.md | 14 +++ CLAUDE.md | 2 +- cmd/mxcli/syntax/features_misc.go | 8 ++ .../reference/navigation/alter-navigation.md | 8 ++ .../doctype-tests/navigation-offline-sync.mdl | 5 ++ mdl/ast/ast_navigation.go | 24 +++-- mdl/backend/modelsdk/navigation_read.go | 25 ++++++ .../modelsdk/navigation_throw_sync_test.go | 89 +++++++++++++++++++ mdl/backend/modelsdk/navigation_write.go | 8 ++ mdl/backend/mpr/convert.go | 1 + mdl/backend/mpr/convert_roundtrip_test.go | 4 +- mdl/executor/cmd_navigation.go | 9 ++ mdl/grammar/MDLParser.g4 | 11 +++ mdl/types/navigation.go | 12 +++ mdl/visitor/visitor_navigation.go | 5 ++ sdk/mpr/parser_misc.go | 7 ++ sdk/mpr/writer_navigation.go | 7 ++ 18 files changed, 228 insertions(+), 12 deletions(-) create mode 100644 mdl/backend/modelsdk/navigation_throw_sync_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index fc1a2e7945..9f98f60b9f 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -85,3 +85,4 @@ {"area": "mdl/backend", "date": "2026-09-08", "symptom": "A property Studio Pro writes on an offline entity config (CompatibilityMode) was read from the model and silently discarded, so any future write path would have dropped it with no error and no mx check failure", "cause": "types.NavOfflineEntity carried three of the four properties Studio Pro actually writes. TestFieldCountDrift, which exists to catch exactly this on hand-copied structs, did not list NavOfflineEntity or NavigationProfile — so adding the field left the guard passing vacuously", "file": "`mdl/types/navigation.go`, `mdl/backend/mpr/convert.go`, `mdl/backend/mpr/convert_roundtrip_test.go`", "insight": "A drift guard is only worth what its list covers, and a guard that passes on a struct it does not know about is worse than none — it reads as coverage. Check the guard names your type before trusting a green run. Also: measure which properties are actually WRITTEN before deciding what to carry — gen declared six here and the reference document had four, with DownloadMode and ShouldDownload occurring zero times, so the risk was inverted from the expected one (writing a property Studio Pro fills in on load, not dropping one)", "refs": ["ako/TestApp", "ako/mxcli#413"]} {"area": "mdl/backend", "date": "2026-09-08", "symptom": "An offline navigation profile authored by mxcli builds, routes and installs as a PWA, and shows an empty app — every gate green", "cause": "MDL had no syntax for offline synchronization, so a created offline profile got an empty OfflineEntityConfigs list. A Mendix offline profile downloads nothing until each entity has a sync mode; `mx check` reports 0 errors either way because an empty list is valid", "file": "`mdl/grammar/MDLParser.g4` (navSyncDef), `mdl/backend/modelsdk/navigation_write.go`, `sdk/mpr/writer_navigation.go`", "insight": "Creating a document kind is not the same as being able to configure it, and the gap is invisible to every static check — the symptom is an empty screen at runtime. When adding a profile/document kind, ask what makes it DO anything, not just what makes it exist. The write is an overlay keyed by entity so CompatibilityMode (stored, unauthorable) survives; building the element from the spec alone would clear it silently, the access-rule defect again", "refs": ["ako/TestApp", "PROPOSAL_offline_sync_configuration.md"]} {"area": "mdl/backend", "date": "2026-09-08", "symptom": "`alter page P { set PageSize = 10 on }` errors `pluggable property \"PageSize\" not found` on a grid that `create page \u2026 (PageSize: 20)` had just written and the app really pages at. `mxcli check --references` passes the script, so it fails only at exec, after earlier statements have landed; DESCRIBE PAGE prints the same capitalised `PageSize:`, so describe \u2192 edit \u2192 exec produced a script mxcli refused to run", "cause": "A pluggable property key is lowerCamel in the widget template (`pageSize`). CREATE resolves the author's spelling case-INsensitively (widget engine `lookupProperty`, and `WidgetV3.GetStringProp` before it); ALTER went through `setPluggableWidgetPropertyMut`, which compared the template key byte-for-byte, so only the exact `pageSize` worked. Direct sequel to Findings #1 (2026-07-27), which fixed the same class for FIRST-CLASS props and deliberately left the pluggable fallback case-sensitive with the comment 'template keys must match the template exactly'", "file": "`mdl/backend/pagemutator/mutator.go` (`setPluggableWidgetPropertyMut`)", "insight": "`strings.EqualFold` against the widget's own PropertyTypes. **The disproven belief is the reusable part**: keys are STORED case-sensitively, which is not a reason to MATCH them that way \u2014 the resolver searches one object type's PropertyTypes, and across every shipped template and definition that scope holds no two keys differing only in case (96 scopes, 1208 keys, 0 collisions). Measure the ambiguity before assuming it; here there was none, and the assumption cost a whole verb. **Cheapest localiser**: run the failing statement with the template's own casing \u2014 `set pageSize` succeeded where `set PageSize` failed, in ONE measurement, on the same widget in the same project. Both engines share `pagemutator`, so an engine split says nothing here (verified: modelsdk and legacy both fixed by the one change). Tests `TestSetPluggableProperty_MatchesTemplateKeyRegardlessOfCase` (+ typo-still-errors control, + `TestPluggablePropertyKeysAreUniqueIgnoringCase` pinning the no-collision argument); repro `mdl-examples/bug-tests/alter-page-pluggable-property-casing.mdl`; verified 0 errors on `mx check` 11.13.0. Two reporter claims did NOT hold: `describe page` DOES emit PageSize, but only when it differs from the widget default 20 (deliberate, so describe round-trips) \u2014 at the default it is omitted, which reads as 'describe cannot show it'. Still open and separate: `check --references` does not resolve pluggable property names at all, so a genuine typo (`PagSize`) still checks clean and fails at exec", "refs": ["mendixlabs/mxcli#1069"]} +{"area": "mdl/backend", "date": "2026-09-09", "symptom": "A control that should have proven a guard was load-bearing PASSED with the guard removed — the test could not distinguish a correct writer from one that reset the property on every write", "cause": "The fixture stored `false` for a boolean property, and false is also the zero value. A writer that ignored the spec entirely wrote false; the correct writer preserved false. Identical output, so the assertion held either way", "file": "`mdl/backend/modelsdk/navigation_throw_sync_test.go`", "insight": "For a BOOLEAN property, a preservation test must exercise BOTH stored values — the non-zero one is the only case that can fail. This is the second time in one feature: CompatibilityMode needed a synthetic `true` because all seven reference configs carry false. The generalisation: when every real document agrees on a value, the fixture drawn from real documents cannot test preservation, and a synthetic counter-case is not optional. The tell is a control that fails to fail — if stubbing the guard leaves the suite green, the test is measuring nothing, and that is worse than no test because it reads as coverage", "refs": ["ako/mxcli#413", "ThrowPartialSyncError"]} diff --git a/.claude/skills/mendix/manage-navigation/SKILL.md b/.claude/skills/mendix/manage-navigation/SKILL.md index 79c713bce1..2c883b84c0 100644 --- a/.claude/skills/mendix/manage-navigation/SKILL.md +++ b/.claude/skills/mendix/manage-navigation/SKILL.md @@ -260,6 +260,20 @@ The `sync` row names the profile. Every mode produces one, **including the modes that download nothing**: a profile with `sync X never` still names `X`, so renaming or dropping it leaves the configuration dangling. +**Errors when the server rejects an object.** Studio Pro's *"Throw error when +server rejects objects during synchronization"* checkbox: + +```sql +create or replace navigation PhoneOffline + home page MyModule.Mobile_Dashboard + on sync error continue; -- default is `throw` +``` + +It uses the phrase MDL already has for failure handling — a microflow's +`on error continue` — rather than a keyword of its own. Omitting the clause +leaves the stored value alone; `describe navigation` emits it only when it is +not the default, so existing scripts stay quiet. + **Compatibility mode has no syntax.** mxcli reads it, preserves it across a rewrite, and `describe navigation` flags any entity that has it on — it is never silently dropped. diff --git a/CLAUDE.md b/CLAUDE.md index e291e9b03d..0433a4818e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -815,7 +815,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati ## Current Implementation Status **Implemented:** -- Offline synchronization (`CREATE NAVIGATION … SYNC (…)`): an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. Both halves are in the catalog: `CATALOG.OFFLINE_ENTITY_CONFIGS` holds one row per configured entity (the profile's `OfflineEntityCount` said how many and nothing else), and a configured entity emits a **`sync` edge** into `CATALOG.REFS` so `show references to Mod.Entity` names the profiles that download it. Every mode gets an edge, **including the ones that download nothing** — a profile with `sync X never` still names X, so renaming or dropping it leaves the config dangling, which is exactly what the edge exists to reveal. See `.claude/skills/mendix/manage-navigation/SKILL.md` and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` +- Offline synchronization (`CREATE NAVIGATION … SYNC (…)`): an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. `ON SYNC ERROR THROW|CONTINUE` writes `ThrowPartialSyncError`, a property **neither generated source declares** (zero occurrences in gen and in generated/metamodel), so it is read from `element.Base.Raw()` and written as a raw key. The spec field is a **pointer**: the property is a bare bool with no unset value, so a non-pointer would reset it on every rewrite that never mentions the clause. Absent reads as **true**, matching every reference profile and Studio Pro's checked-by-default box. Both halves are in the catalog: `CATALOG.OFFLINE_ENTITY_CONFIGS` holds one row per configured entity (the profile's `OfflineEntityCount` said how many and nothing else), and a configured entity emits a **`sync` edge** into `CATALOG.REFS` so `show references to Mod.Entity` names the profiles that download it. Every mode gets an edge, **including the ones that download nothing** — a profile with `sync X never` still names X, so renaming or dropping it leaves the config dangling, which is exactly what the edge exists to reveal. See `.claude/skills/mendix/manage-navigation/SKILL.md` and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` - Project brain (`mxcli brain init/capture/staged/promote/drop/check/show`): an **opt-in** store in `docs/brain/` for the project knowledge mxcli cannot compute. The governing rule is that anything derivable from the model is answered by a command and never written down — a note that transcribes the model disagrees with it silently. Records shard by **anchor scope**: an entry's first anchor names its file (`@Sales.Order` → `modules/Sales.md`), an anchorless entry is cross-cutting (`project.md`), and there is no index to maintain because the module prefix *is* the file name. That is what makes the cap per-shard rather than a project-wide budget, and lets a session load `project.md` plus the modules it is touching. `check` answers two independent questions: each anchor is **resolved / not found / not indexable** — only the middle one fails, and the third exists because the catalog's `objects` view covers the describable types only, so a scheduled event would otherwise read as *missing* (separated with `FindDocumentUnit`, which cannot miss a kind because it never asks what kind anything is). Misfiling is a **second axis, not a fourth state**: every anchor can resolve and the entry still be in the wrong file, and it is only decided when something resolved — judging it on an all-not-indexable entry reintroduced the same false staleness through the other axis (caught by a test, with the guard stubbed as the control). An agent `capture`s to a git-ignored queue and a person `promote`s; the queue is deliberately **not** sharded, because routing it would force the file decision before a human has looked at the entry. `mxcli lint` prints the unpromoted-queue count, because a report only `brain check` prints is a report nothing demands. Sizes are computed by `brain show` and never written into a committed file. A second record kind, **requirement**, lives in `plan/.md` and inverts the anchor's meaning: a decision's anchor points backward (not resolving = stale, fails), a requirement's points forward (not resolving = not built yet, passes). Measured: filed as an ordinary entry, one unbuilt requirement takes `brain check` to exit 1 — which is why it is a separate kind rather than more entries in the same files. That inversion is also what makes `brain plan` a real progress report: a requirement is *built* when its anchors resolve, so creating the microflow it names moves the count with the plan file untouched (measured 0/1 → 1/0). A status written beside a requirement is therefore refused by the skill, not just discouraged. Slices are ordered by name (`01-accounts`), span modules by design (so misfiling does not apply), and carry a generous cap that enforces the slicing discipline — a slice too long to read should be split. A third kind, **open question** (`--open`), records what is *not* decided; its anchors are deliberately **not** checked, since the question is often whether the thing should exist at all — measured, the identical anchor exits 1 as a decision and 0 as a question. `brain resolve` converts one into a decision in place, keeping its id and position and starting to check its anchors, which is the transition the kind exists for. Unanswered questions are reported by `brain check` and by `mxcli lint`. The skill also gives capture a **trigger** rather than good intentions — a correction you have had to make twice — because the decisions half otherwise under-fills while the plan half fills at bootstrap. `bootstrap-app` asks for requirements at the interview and records them by default. Package: `cmd/mxcli/brain/`. See `docs-site/src/tools/project-brain.md` and `docs/11-proposals/PROPOSAL_project_brain.md` - Default styling + runtime theme switching (`mxcli theme list/show/create/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). A project can add its own themes under `theme/mxcli-themes//` (committed, not compiled); `theme create [--from ]` scaffolds one from an existing theme, renaming the identifiers built from the name and optionally seeding the palette from `--mxt-*` declarations in a design artifact. A local theme shadows a built-in of the same name. Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` - MPR v1/v2 reading and writing diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index b1cc2d3efe..9dc32acdf4 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -154,6 +154,7 @@ DISCONNECT;`, "navigation profile", "phone profile", "tablet profile", "offline profile", "offline navigation", "sync", "synchronization", "offline sync", "offline entity", "pwa", "download mode", + "throw error", "sync error", "partial sync", "server rejects", }, Syntax: `CREATE OR REPLACE NAVIGATION HOME PAGE Module.Page @@ -164,6 +165,7 @@ DISCONNECT;`, MENU ITEM 'Label' PAGE Module.Page [ICON Module.IconCollection.Name]; MENU 'Group' [ICON Module.IconCollection.Name] ( ... ); )] + [ON SYNC ERROR THROW|CONTINUE] [SYNC ( SYNC Module.Entity ONLINE; SYNC Module.Entity ALL; @@ -214,6 +216,12 @@ DISCONNECT;`, -- and a stored constraint already carries Mendix's own escaping, so the two -- compose into runs of six quotes. DESCRIBE emits the bracket form. -- +-- ON SYNC ERROR is Studio Pro's "Throw error when server rejects objects +-- during synchronization", and defaults to THROW. It uses the phrase MDL +-- already has for failure handling (a microflow's ON ERROR CONTINUE) rather +-- than a new keyword. OMITTING it leaves the stored value alone; DESCRIBE emits +-- it only when it is not the default. +-- -- The block REPLACES the stored list, the way MENU replaces the menu. An -- entity's compatibility-mode flag has no syntax and is preserved across the -- rewrite untouched; DESCRIBE NAVIGATION flags it rather than dropping it. diff --git a/docs-site/src/reference/navigation/alter-navigation.md b/docs-site/src/reference/navigation/alter-navigation.md index 4f84cd32de..a416715e22 100644 --- a/docs-site/src/reference/navigation/alter-navigation.md +++ b/docs-site/src/reference/navigation/alter-navigation.md @@ -11,6 +11,7 @@ CREATE OR REPLACE NAVIGATION profile [ MENU ( menu_items ) ] + [ ON SYNC ERROR { THROW | CONTINUE } ] [ SYNC ( sync_rules ) ] @@ -115,6 +116,13 @@ quoted `WHERE 'xpath'` still parses, but every quote inside it must be doubled. The block replaces the stored list, the way `MENU` replaces the menu. Omitting it leaves the stored configuration alone. +`ON SYNC ERROR THROW | CONTINUE` is Studio Pro's *"Throw error when server +rejects objects during synchronization"*, and defaults to `THROW`. It reuses the +phrase MDL already has for failure handling — a microflow's `ON ERROR CONTINUE` +— rather than introducing a keyword of its own. Omitting the clause leaves the +stored value alone, and `DESCRIBE NAVIGATION` emits it only when it is not the +default. + An entity's *compatibility mode* flag has no MDL syntax. It is read, preserved across a rewrite, and reported by `DESCRIBE NAVIGATION` — never silently dropped. diff --git a/mdl-examples/doctype-tests/navigation-offline-sync.mdl b/mdl-examples/doctype-tests/navigation-offline-sync.mdl index 9ced3e3ee1..e0fb918010 100644 --- a/mdl-examples/doctype-tests/navigation-offline-sync.mdl +++ b/mdl-examples/doctype-tests/navigation-offline-sync.mdl @@ -51,6 +51,11 @@ create page "OfflineSync"."Mobile_Home" -- names plus "Offline". create or replace navigation "PhoneOffline" home page "OfflineSync"."Mobile_Home" + -- Studio Pro's "Throw error when server rejects objects during + -- synchronization". Spelled with the phrase MDL already uses for failure + -- handling (a microflow's ON ERROR CONTINUE), and omitting the clause leaves + -- the stored value alone rather than resetting it. + on sync error continue sync ( -- Fetched from the server; never held on the device. sync "OfflineSync"."Setting" online; diff --git a/mdl/ast/ast_navigation.go b/mdl/ast/ast_navigation.go index 001388e9f1..352508efdf 100644 --- a/mdl/ast/ast_navigation.go +++ b/mdl/ast/ast_navigation.go @@ -7,15 +7,21 @@ import "github.com/mendixlabs/mxcli/mdl/types" // AlterNavigationStmt represents: CREATE [OR REPLACE] NAVIGATION [clauses...] // This is a full-replacement command: omitted clauses clear that section. type AlterNavigationStmt struct { - ProfileName string // e.g. "Responsive" - HomePages []NavHomePageDef // HOME PAGE/MICROFLOW ... [FOR role] - LoginPage *QualifiedName // LOGIN PAGE ... - NotFoundPage *QualifiedName // NOT FOUND PAGE ... - MenuItems []NavMenuItemDef // MENU (...) block - HasMenuBlock bool // true if MENU (...) was present (even if empty → clears menu) - SyncEntries []NavSyncDef // SYNC (...) block — offline synchronization - HasSyncBlock bool // true if SYNC (...) was present (even if empty → clears the list) - CreateOrModify bool // true if CREATE OR REPLACE/MODIFY was used + ProfileName string // e.g. "Responsive" + HomePages []NavHomePageDef // HOME PAGE/MICROFLOW ... [FOR role] + LoginPage *QualifiedName // LOGIN PAGE ... + NotFoundPage *QualifiedName // NOT FOUND PAGE ... + MenuItems []NavMenuItemDef // MENU (...) block + HasMenuBlock bool // true if MENU (...) was present (even if empty → clears menu) + SyncEntries []NavSyncDef // SYNC (...) block — offline synchronization + HasSyncBlock bool // true if SYNC (...) was present (even if empty → clears the list) + // ThrowSyncError is ON SYNC ERROR THROW|CONTINUE, and is a POINTER so an + // omitted clause leaves the stored value alone. A plain bool would make + // every rewrite that never mentions it reset the flag to false — the + // guard-don't-drop failure, in the one property on this statement that is + // a bare boolean and so has no "unset" value of its own. + ThrowSyncError *bool + CreateOrModify bool // true if CREATE OR REPLACE/MODIFY was used } func (s *AlterNavigationStmt) isStatement() {} diff --git a/mdl/backend/modelsdk/navigation_read.go b/mdl/backend/modelsdk/navigation_read.go index 5c3cd6ac5b..b05e9dfc4c 100644 --- a/mdl/backend/modelsdk/navigation_read.go +++ b/mdl/backend/modelsdk/navigation_read.go @@ -95,6 +95,7 @@ func webNavProfileFromGen(p *genNav.NavigationProfile) *types.NavigationProfile } } appendOfflineEntities(profile, p.OfflineEntityConfigsItems()) + profile.ThrowPartialSyncError = throwPartialSyncError(p.Raw()) return profile } @@ -342,3 +343,27 @@ func textOf(el element.Element) string { } return "" } + +// throwPartialSyncError reads a property NEITHER modelsdk/gen NOR +// generated/metamodel declares — measured on ako/TestApp, zero occurrences in +// each — so there is no typed accessor to call. element.Base keeps the raw +// document, which is what makes reading it possible without a second load. +// +// Absent means TRUE: every reference profile carries true and Studio Pro's box +// is checked by default, so a document without the key must not be read as +// "do not throw". +// +// Web profiles only. Both of ako/TestApp's carry it; whether a native profile +// does is unmeasured, and nativeNavProfileFromGen is deliberately left alone +// rather than given a default nothing has verified. +func throwPartialSyncError(raw bson.Raw) bool { + v, err := raw.LookupErr("ThrowPartialSyncError") + if err != nil { + return true + } + b, ok := v.BooleanOK() + if !ok { + return true + } + return b +} diff --git a/mdl/backend/modelsdk/navigation_throw_sync_test.go b/mdl/backend/modelsdk/navigation_throw_sync_test.go new file mode 100644 index 0000000000..2ecff36d29 --- /dev/null +++ b/mdl/backend/modelsdk/navigation_throw_sync_test.go @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" + // This package uses BOTH driver majors: navigation_write.go is on v1 and + // navigation_read.go on v2. The aliases are what keep a test that spans + // the write and read paths from silently mixing them. + bsonv1 "go.mongodb.org/mongo-driver/bson" + bsonv2 "go.mongodb.org/mongo-driver/v2/bson" +) + +func boolPtr(b bool) *bool { return &b } + +func throwOf(t *testing.T, doc bsonv1.D) any { + t.Helper() + for _, e := range doc { + if e.Key == "ThrowPartialSyncError" { + return e.Value + } + } + return nil +} + +// The spec field is a POINTER, and this is why. ThrowPartialSyncError is a bare +// bool with no unset value of its own, so a non-pointer spec would make every +// rewrite that never mentions the clause reset the stored flag — silently, on a +// property neither generated source declares and nothing else reports. +func TestThrowSyncErrorIsLeftAloneWhenTheStatementIsSilent(t *testing.T) { + // BOTH stored values, because false is also the zero value: a writer that + // reset the flag on every rewrite would be indistinguishable from a correct + // one if the fixture only ever stored false. Same trap as CompatibilityMode, + // and it was found by a control that failed to fail. + for _, storedValue := range []bool{true, false} { + stored := bsonv1.D{ + {Key: "Name", Value: "TabletOffline"}, + {Key: "ThrowPartialSyncError", Value: storedValue}, + } + out := navPatchWebProfile(stored, types.NavigationProfileSpec{}) + if got := throwOf(t, out); got != storedValue { + t.Errorf("a rewrite that never mentioned the clause changed a stored %v to %v", + storedValue, got) + } + } + + stored := bsonv1.D{ + {Key: "Name", Value: "TabletOffline"}, + {Key: "ThrowPartialSyncError", Value: false}, + } + + // Control in the other direction: when the statement DOES say something, + // it must actually be written — otherwise the first assertion passes for + // a writer that ignores the field entirely. + out := navPatchWebProfile(stored, types.NavigationProfileSpec{ThrowSyncError: boolPtr(true)}) + if got := throwOf(t, out); got != true { + t.Errorf("`on sync error throw` did not write: %v", got) + } + out = navPatchWebProfile(stored, types.NavigationProfileSpec{ThrowSyncError: boolPtr(false)}) + if got := throwOf(t, out); got != false { + t.Errorf("`on sync error continue` did not write: %v", got) + } +} + +// Absent must read as true, not as the zero value. Every reference profile +// carries true and Studio Pro's box is checked by default, so defaulting to +// false would silently turn off error reporting for a document that simply +// predates the key. +func TestAbsentThrowPartialSyncErrorReadsAsTrue(t *testing.T) { + empty, err := bsonv2.Marshal(bsonv2.D{{Key: "Name", Value: "Responsive"}}) + if err != nil { + t.Fatal(err) + } + if !throwPartialSyncError(bsonv2.Raw(empty)) { + t.Error("an absent key must read as true") + } + + // Control: a present false is honoured, so the default is not masking the + // read. + set, err := bsonv2.Marshal(bsonv2.D{{Key: "ThrowPartialSyncError", Value: false}}) + if err != nil { + t.Fatal(err) + } + if throwPartialSyncError(bsonv2.Raw(set)) { + t.Error("a stored false must be read as false") + } +} diff --git a/mdl/backend/modelsdk/navigation_write.go b/mdl/backend/modelsdk/navigation_write.go index 6e0d3cccd3..891b44dbe5 100644 --- a/mdl/backend/modelsdk/navigation_write.go +++ b/mdl/backend/modelsdk/navigation_write.go @@ -202,6 +202,14 @@ func navPatchWebProfile(doc bson.D, spec types.NavigationProfileSpec) bson.D { doc = navSetField(doc, "OfflineEntityConfigs", navOfflineConfigs(navGetArray(doc, "OfflineEntityConfigs"), spec.OfflineEntities)) } + + // ThrowPartialSyncError is declared by neither generated source, so it can + // only be written as a raw key — which is why nil means "the statement said + // nothing" and the stored value is left exactly as it is. A non-pointer + // would reset the flag on every rewrite that never mentions it. + if spec.ThrowSyncError != nil { + doc = navSetField(doc, "ThrowPartialSyncError", *spec.ThrowSyncError) + } return doc } diff --git a/mdl/backend/mpr/convert.go b/mdl/backend/mpr/convert.go index 7e69963b1f..77a661ea38 100644 --- a/mdl/backend/mpr/convert.go +++ b/mdl/backend/mpr/convert.go @@ -220,6 +220,7 @@ func convertNavProfile(in *mpr.NavigationProfile) *types.NavigationProfile { p.MenuItems[i] = convertNavMenuItem(mi) } } + p.ThrowPartialSyncError = in.ThrowPartialSyncError if in.OfflineEntities != nil { p.OfflineEntities = make([]*types.NavOfflineEntity, len(in.OfflineEntities)) for i, oe := range in.OfflineEntities { diff --git a/mdl/backend/mpr/convert_roundtrip_test.go b/mdl/backend/mpr/convert_roundtrip_test.go index 7c19838c81..0bb0432515 100644 --- a/mdl/backend/mpr/convert_roundtrip_test.go +++ b/mdl/backend/mpr/convert_roundtrip_test.go @@ -632,8 +632,8 @@ func TestFieldCountDrift(t *testing.T) { // CompatibilityMode to NavOfflineEntity left this test passing while the // new field was silently not carried, which is the exact drift the test // exists to catch. - assertFieldCount(t, "mpr.NavigationProfile", mpr.NavigationProfile{}, 9) - assertFieldCount(t, "types.NavigationProfile", types.NavigationProfile{}, 9) + assertFieldCount(t, "mpr.NavigationProfile", mpr.NavigationProfile{}, 10) + assertFieldCount(t, "types.NavigationProfile", types.NavigationProfile{}, 10) assertFieldCount(t, "mpr.NavOfflineEntity", mpr.NavOfflineEntity{}, 4) assertFieldCount(t, "types.NavOfflineEntity", types.NavOfflineEntity{}, 4) } diff --git a/mdl/executor/cmd_navigation.go b/mdl/executor/cmd_navigation.go index a53d52f9a6..2dc203b633 100644 --- a/mdl/executor/cmd_navigation.go +++ b/mdl/executor/cmd_navigation.go @@ -96,6 +96,7 @@ func execAlterNavigation(ctx *ExecContext, s *ast.AlterNavigationStmt) error { spec.MenuItems = append(spec.MenuItems, convertMenuItemDef(mi)) } + spec.ThrowSyncError = s.ThrowSyncError spec.HasSync = s.HasSyncBlock for _, se := range s.SyncEntries { spec.OfflineEntities = append(spec.OfflineEntities, types.NavOfflineEntitySpec{ @@ -350,6 +351,14 @@ func outputNavigationProfile(ctx *ExecContext, p *types.NavigationProfile) { fmt.Fprintln(ctx.Output, " )") } + // Only emitted when it differs from the platform default, so the clause + // appears exactly when it carries information. Describing every profile + // with `on sync error throw` would add a line to every navigation script + // that says what would happen anyway. + if !p.ThrowPartialSyncError { + fmt.Fprintln(ctx.Output, " on sync error continue") + } + // Offline entities. These are re-executable now, so they are emitted as a // SYNC block rather than as the commented-out approximation that made // describe -> exec lossy for every project using offline sync. diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index ec4265c58f..ac1d861121 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -331,6 +331,17 @@ navigationClause | NOT FOUND PAGE qualifiedName | MENU_KW LPAREN navMenuItemDef* RPAREN | SYNC LPAREN navSyncDef* RPAREN + // Studio Pro's "Throw error when server rejects objects during + // synchronization", stored as the profile-level ThrowPartialSyncError. + // + // Spelled with the phrase MDL already uses for failure handling — a + // microflow's ON ERROR CONTINUE / ON ERROR ROLLBACK — so it needs no new + // token and reads as something already learned. "Reject" is the platform's + // own word, but REJECT appears ~500 times across the examples and skills + // (approve/reject is one of the commonest things a workflow models), and + // claiming a heavily-used identifier as a keyword is not worth the closer + // paraphrase. + | ON SYNC ERROR (THROW | CONTINUE) ; // Offline synchronization, one statement per entity, mirroring the MENU block: diff --git a/mdl/types/navigation.go b/mdl/types/navigation.go index 55902136c9..eccc0c438e 100644 --- a/mdl/types/navigation.go +++ b/mdl/types/navigation.go @@ -33,6 +33,12 @@ type NavigationProfile struct { NotFoundPage string `json:"notFoundPage,omitempty"` MenuItems []*NavMenuItem `json:"menuItems,omitempty"` OfflineEntities []*NavOfflineEntity `json:"offlineEntities,omitempty"` + // ThrowPartialSyncError is stored on every WEB profile, online ones + // included, and is declared by NEITHER modelsdk/gen NOR + // generated/metamodel — measured on ako/TestApp, zero occurrences in each. + // It is therefore read and written as raw BSON rather than through the + // codec's typed accessors. + ThrowPartialSyncError bool `json:"throwPartialSyncError,omitempty"` } // NavHomePage holds a profile's default home page. @@ -201,6 +207,12 @@ type NavigationProfileSpec struct { // field alone is not enough. OfflineEntities []NavOfflineEntitySpec HasSync bool + // ThrowSyncError is Studio Pro's "Throw error when server rejects objects + // during synchronization". A POINTER, so nil means the statement said + // nothing and the stored value is left alone — the property is a bare bool + // with no unset value of its own, so a non-pointer would silently reset it + // on every rewrite. + ThrowSyncError *bool } // NavOfflineEntitySpec is one entity's offline sync rule, as MDL can express diff --git a/mdl/visitor/visitor_navigation.go b/mdl/visitor/visitor_navigation.go index 4e24c33ce0..13a83ad7d0 100644 --- a/mdl/visitor/visitor_navigation.go +++ b/mdl/visitor/visitor_navigation.go @@ -84,6 +84,11 @@ func (b *Builder) processNavigationClause(stmt *ast.AlterNavigationStmt, ctx *pa item := buildNavMenuItemDef(itemCtx) stmt.MenuItems = append(stmt.MenuItems, item) } + } else if ctx.ON() != nil && ctx.SYNC() != nil && ctx.ERROR() != nil { + // ON SYNC ERROR THROW|CONTINUE. Checked before the bare SYNC block + // because both alternatives carry a SYNC token. + throw := ctx.THROW() != nil + stmt.ThrowSyncError = &throw } else if ctx.SYNC() != nil { // SYNC (navSyncDef*) stmt.HasSyncBlock = true diff --git a/sdk/mpr/parser_misc.go b/sdk/mpr/parser_misc.go index 16f3e21a7c..690140df72 100644 --- a/sdk/mpr/parser_misc.go +++ b/sdk/mpr/parser_misc.go @@ -546,6 +546,13 @@ func parseNavigationProfile(raw map[string]any) *NavigationProfile { } } + // Studio Pro writes this on every web profile, online ones included, and + // neither gen nor generated/metamodel declares it — so it is read straight + // off the raw document. Defaulting to true matches every reference profile + // and Studio Pro's own checked-by-default box, so a document that somehow + // lacks the key is not silently flipped to "do not throw". + profile.ThrowPartialSyncError = extractBool(raw["ThrowPartialSyncError"], true) + // Offline entity configs (both web and native) for _, item := range extractBsonArray(raw["OfflineEntityConfigs"]) { if oeMap, ok := item.(map[string]any); ok { diff --git a/sdk/mpr/writer_navigation.go b/sdk/mpr/writer_navigation.go index 3efdd68fb3..317322d738 100644 --- a/sdk/mpr/writer_navigation.go +++ b/sdk/mpr/writer_navigation.go @@ -200,6 +200,13 @@ func patchWebProfile(doc bson.D, spec NavigationProfileSpec) bson.D { buildOfflineConfigsBson(getBsonArray(doc, "OfflineEntityConfigs"), spec.OfflineEntities)) } + // Kept identical to the modelsdk engine: nil leaves the stored flag alone, + // because neither generated source declares the property and a non-pointer + // would reset it on every rewrite that never mentions it. + if spec.ThrowSyncError != nil { + doc = setBsonField(doc, "ThrowPartialSyncError", *spec.ThrowSyncError) + } + return doc } From 8db3cb5cff4270aae8a8d5f23f1ddae2afbaa397 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 08:03:32 +0000 Subject: [PATCH 07/19] fix(widgets): never drop a visibility rule the older vocabulary already lifted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conjunction handling withheld any rule it could not read in full. That is right for a guard shape this extractor never supported before — emitting one conjunct of a nested condition over-fires, as `!1===e.showNumberOfRows` alone does for Datagrid's pagingPosition. It is wrong for a rule the older, single-condition vocabulary already produced: that rule's accuracy is unchanged by conjunction support, and dropping it loses detection the previous release had. Six of Combo box's rules were exactly that shape and had gone. The measurement missed it. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` — 33 of them — but `widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42, and the 9 missing included the widget the whole change was justified by. Re-measured over all 42: 133 -> 157 of 237 recognized (56% -> 66%), 28 rules gained, zero lost, zero per-widget regressions, and two bindings removed from generated examples (Combo box's filterInputDebounceInterval under an unbound onChange event, TreeNode's openNodeOn), both traced to source. The regression test takes the real Combo box .mpk rather than a reconstruction. A hand-written snippet of the same apparent shape passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and proved nothing. Against the real input the control fails naming all six rules. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KTdvGwVewkdgNQGDCxpQPZ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + mdl/executor/editorconfig_extract.go | 36 +++++++- mdl/executor/editorconfig_shapes_test.go | 85 +++++++++++++++++++ 3 files changed, 119 insertions(+), 3 deletions(-) diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index e2d3ad9c33..267a74f52b 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -567,3 +567,4 @@ {"area": "mdl/executor", "date": "2026-09-08", "refs": ["mendixlabs/mxcli#1073", "mendixlabs/mxcli#1020"], "ce": ["CE7252"], "symptom": "A `call external action` on an OData action that has a NULLABLE parameter is CE7252 \"The parameters for remote action '' have changed\", with no MDL that clears it. Reported as a missing syntax for an empty/null binding: `= null`, `= empty`, a bare `= )` and omitting the parameter were all tried. Persists after upgrading past the #1020 fix, because it is a different missing field behind the same CE code.", "file": "mdl/executor/cmd_microflows_builder_calls.go (externalParamKind.canBeEmpty, paramCanBeEmpty, resolveExternalActionParameterKinds, addCallExternalActionAction); engine-agnostic - the fix is in the shared semantic builder, so both modelsdk and legacy are covered by one change", "cause": "Microflows$ExternalActionParameterMapping.CanBeEmpty was never set, so it was Go's false on every mapping mxcli wrote. Mendix compares it against the contract's Nullable on every build and reports the disagreement as CE7252. mdl/types.EdmActionParameter ALREADY parsed Nullable as a three-state *bool; the value was simply dropped between the parser and the mapping builder, so the fix is to carry it - no new parsing. The default is the subtle half: CSDL makes Nullable optional on and defaults it to TRUE, the opposite of Go's zero value.", "insight": "**The reported premise was wrong and the bug was real; they were not the same thing.** A Studio Pro reference document settled both at once: on ako/TestApp 11.14.0 the mappings are {command, Argument \"empty\", CanBeEmpty false} and {additional, Argument \"empty\", CanBeEmpty true}. So (a) `Argument` is the EXPRESSION `empty`, never an empty string - an unfilled argument in Studio Pro is the Mendix null literal, so `additional = empty` was always correct MDL and already wrote a byte-identical Argument; and (b) the thing that actually differed was CanBeEmpty, which no MDL syntax reaches because it is derived from the contract, not typed by the developer. **Two hypotheses died on that one document**: that DESCRIBE's `additional = )` output (real - formatAction appends unconditionally where the java-action branch guards on empty) was what users hit, and that the empty-Argument state was reachable at all. It is not: DESCRIBE round-trips the real document correctly as `command = empty, additional = empty`. **Do not attribute a CE code to the last bug that produced it.** #1020 produced CE7252 from a missing ParameterType and was fixed in v0.21.0, which made \"upgrade\" look like the answer; the same code from a different field on HEAD was only found by building the reporter's shape and running mxbuild. **Verify version claims in the ISSUE against the grammar, not the changelog**: the three rules involved (callArgument, literal, callExternalActionStatement) are byte-identical at v0.20.0, so the reported parse errors for `= null`/`= empty` never happened at any version - an unrelated error elsewhere in the script (a missing microflow parameter list produces `missing '(' at 'begin'`) reads as an argument error and cost a probe here too. **Controls**: reverting only `mapping.CanBeEmpty = pk.canBeEmpty` takes the 4-statement repro from 0 to 4 errors, one CE7252 per call; separately, defaulting an ABSENT Nullable to false (rather than true) reproduces CE7252 on the Annotate action alone, which is what pins the CSDL default. Studio Pro's own microflow in the same project is the 0-error control. **Left unfixed on purpose**: Studio Pro also writes empty marker arrays AdditionalAttributes and IncludedAssociations (marker 2) on the call and each mapping; mxbuild 11.14 builds at 0 errors without them and they are unverified against Studio Pro, so they are recorded rather than guessed at. Repro mdl-examples/bug-tests/odata-1073-external-action-nullable-params.mdl"} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "`mxcli widget describe` over-listed required bindings because half of each widget's editor visibility rules were never lifted: Combo box reported '16 of 32 editor hide-rules recognized', so properties its editor hides in the described configuration were still asked for. Across the 25 widgets in the fixture that carry rules, 99 of 177 hide-calls (56%) were recognized.", "cause": "Two independent gaps. (1) The guard vocabulary covered only string equality and bare truthiness, so `null===e.dataSource`, minified booleans `!1===e.showFooter`, `0===e.list.length` and `[\"a\",\"b\"].includes(e.type)` all read as unsupported; and a hide that was not FIRST in a `cond ? (hide(a), hide(b))` comma group got no guard at all. (2) The rule model held ONE condition, but editorConfig nests branches — Combo box reaches `source==\"context\" && optionsSourceType==\"association\" && showFooter==false` three levels deep — so a lifted rule could only ever be one conjunct.", "file": "mdl/executor/editorconfig_extract.go, mdl/types/widget_visibility.go", "insight": "Coverage is the wrong thing to optimise on its own: the first cut took Combo box 16->27 and left the describe output BYTE-IDENTICAL, because every consumer skips a rule whose condition value is indeterminable and `exampleValues` records a property only when its default is non-empty — which an unset datasource never is. Moving a metric without moving the outcome is the failure mode to watch for; measure the user-visible artifact (the generated MDL example), not the counter. Worse, several of those 11 new rules were WRONG: they stated one conjunct of a nested condition, and `pagingPosition` (Datagrid) then read as hidden whenever the row count is off, pagination or not — pagination defaults ON, so that is the common case. The fix is to carry the whole conjunction (WidgetVisibilityRule.And) rather than to reject or to guess, and to make every consumer evaluate it (Rule.Fires) — a reader that looks only at HiddenWhen silently over-fires, which is exactly the bug wearing the right field name. Three sub-traps, each found only by diffing rules against the editorConfig source: the innermost enclosing group is often the very group whose guard the hide already carries, so terms must be deduped; the comma walk must not cross a comma that is not a group separator (`A?B:hide(x),hide(y)` gave Maps' `advanced` a guard belonging to a different property); and the PLATFORM argument is not a configuration term — TreeNode gates on `\"web\"===platform`, which MDL always satisfies, so folding it away keeps three bindings correctly pruned. Discipline that made this safe: the acceptance bar was 'zero rules lost, zero regressions, and every removed binding traced to the editorConfig source', not 'the number went up'.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "CI build-and-test fails on a newly added mdl-examples/doctype-tests/ script with `Execution error: ... needs the modelsdk engine (run without MXCLI_ENGINE=legacy)` — while `mxcli check` and a local exec both pass", "cause": "TestMxCheck_DoctypeScripts runs every doctype script through exec + mx check on BOTH engines. A script using a modelsdk-only capability (creating a navigation profile, menu/rule/layout authoring) cannot pass on legacy, where the backend refuses by design rather than approximating the document", "file": "`mdl/executor/roundtrip_doctype_test.go` (engineScriptSkip)", "insight": "A doctype example is a two-engine test, not a one-engine one, and nothing local tells you: `mxcli check` needs no engine and a local exec uses the default (modelsdk). Before adding an example, ask whether anything in it is modelsdk-only — the refusals are deliberate and listed in mdl/backend/mpr/backend.go. The remedy is an engineScriptSkip entry naming WHY the engine refuses, not weakening the script; and note separately whether the feature under test is itself dual-engine, since here only the profile CREATION was modelsdk-only while the SYNC block works on both and is unit-tested on each", "refs": ["ako/mxcli#420"]} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "A widget-describe change measured as 'zero rules lost' had in fact dropped SIX visibility rules from Combo box — the widget the work was justified by. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` (33 of them) but `mxcli widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42; the missing 9 included the one that mattered.", "cause": "The sweep's widget list was derived from an artifact of the pipeline (generated def.json files) rather than from the capability under test (what `widget describe` can describe). Nothing compared the two lists, so the sample silently excluded a whole class — the same 'two lists, nothing comparing them' shape as the defects being fixed.", "file": "mdl/executor/editorconfig_extract.go, mdl/executor/editorconfig_shapes_test.go", "insight": "Derive a sweep's population from the CAPABILITY, never from a convenient artifact, and state the population in the claim: 'zero rules lost across the 33 widgets carrying a def.json' would have been true and would have invited the question. A second trap immediately after: the regression test written to lock the fix used a hand-written editorConfig snippet of the same APPARENT shape, and it passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and the test proved nothing. Pointing the test at the real committed .mpk made the control fail with all six names. Rule of thumb: when a defect was found in real vendor input, the regression test takes the real input; a reconstructed minimal case must be shown to fail without the fix BEFORE it is trusted, and here it did not. The fix itself is the policy that should have been there from the start: conjunction support may WITHHOLD a rule the extractor never produced before (emitting one conjunct over-fires), but may never drop one the older vocabulary already lifted — that rule's accuracy is unchanged by the new work.", "refs": ["mendixlabs/mxcli#1036"]} diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go index 44f0965c67..09fd288b60 100644 --- a/mdl/executor/editorconfig_extract.go +++ b/mdl/executor/editorconfig_extract.go @@ -178,9 +178,23 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit // (…, "openStreet"===B.mapProvider && hide([apiKey, apiKeyExp]))`. default: // The chain could not be read in full, so the terms cannot be - // stored. Keep the rule only for the keys the ternary's other - // branch hides anyway — there the single condition can - // under-report but never claim hidden where the editor shows it. + // stored and the rule states one conjunct of a larger condition. + // + // That is a reason to withhold a rule this extractor did not + // previously produce — emitting it would ADD an over-firing rule, + // as `!1===e.showNumberOfRows` alone does for Datagrid's + // pagingPosition. It is NOT a reason to drop a rule that the + // older, single-condition vocabulary already lifted: that rule's + // accuracy is unchanged by this work, and removing it would lose + // detection the previous release had. Combo box is where that + // distinction shows — six of its rules are exactly this shape. + // + // So: newly-supported guard shapes must earn their place (the + // other branch has to hide the property anyway, which makes the + // single condition safe); shapes that already worked are kept. + if !isNewlySupportedGuard(guardText) { + break + } kept := keys[:0:0] for _, k := range keys { if hiddenInComplementaryBranch(js, callStart, k) { @@ -1058,6 +1072,22 @@ func dedupeConditions(own types.WidgetVisibilityCondition, cs []types.WidgetVisi return out } +// isNewlySupportedGuard reports whether a guard is one of the shapes this +// extractor learned alongside conjunctions — `null===x`, a minified boolean, +// `0===x.length`, `["a","b"].includes(x)`. Those had no rule before, so +// withholding one under a conjunction loses nothing; the older shapes did, and +// withholding theirs would be a regression. +func isNewlySupportedGuard(guard string) bool { + for _, re := range []*regexp.Regexp{ + nullCmpRE, nullCmpRE2, lenCmpRE, lenCmpRE2, boolCmpRE, boolCmpRE2, includesRE, + } { + if re.MatchString(guard) { + return true + } + } + return false +} + // condsSig renders conditions into a dedupe key. func condsSig(cs []types.WidgetVisibilityCondition) string { var b strings.Builder diff --git a/mdl/executor/editorconfig_shapes_test.go b/mdl/executor/editorconfig_shapes_test.go index e72cea5b32..9b8ac0b56f 100644 --- a/mdl/executor/editorconfig_shapes_test.go +++ b/mdl/executor/editorconfig_shapes_test.go @@ -6,6 +6,8 @@ import ( "strings" "testing" + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" + "github.com/mendixlabs/mxcli/mdl/types" ) @@ -241,3 +243,86 @@ func TestRuleTextJoinsEveryTerm(t *testing.T) { } } } + +// A conjunction the walk cannot read is a reason to WITHHOLD a rule this +// extractor did not previously produce — emitting one states a single conjunct +// and over-fires. It is not a reason to drop a rule the older, single-condition +// vocabulary already lifted: that rule's accuracy is unchanged by conjunction +// support, and removing it loses detection the previous release had. +// +// This is the guarantee that failed silently once. The sweep that claimed "zero +// rules lost" listed widgets from their .def.json files, and Combo box — the +// widget the work was justified by — has none, so it was never measured. Six of +// its rules had been dropped. +func TestConjunctionNeverDropsAPreviouslyLiftedRule(t *testing.T) { + // Combo box's REAL editorConfig, not a synthetic stand-in. A hand-written + // snippet of the same apparent shape did NOT reproduce this — it was never + // flagged conjunctive, so the test passed with the fix reverted and proved + // nothing. The nesting that triggers it is three levels deep and specific. + js, err := mpk.ReadEditorConfig( + "../../testdata/expr-checker/widgets/com.mendix.widget.web.Combobox.mpk", + "com.mendix.widget.web.combobox.Combobox") + if err != nil || js == "" { + t.Fatalf("read Combo box editorConfig: %v", err) + } + + rules, _ := extractVisibilityRulesFromJS(js) + got := map[string]bool{} + for _, r := range rules { + got[r.PropertyKey] = true + } + + // Each of these was lifted by the older, single-condition vocabulary and sits + // inside a group whose guard is about the same object. Conjunction support + // must not cost them: their accuracy is unchanged by this work, and dropping + // them loses detection the previous release had. + // + // This guarantee failed once, silently. The sweep that claimed "zero rules + // lost" enumerated widgets from their .def.json files, and Combo box has + // none — it is described straight from the .mpk — so the widget the work was + // justified by was never in the sample. Six rules had gone. + for _, want := range []string{ + "databaseAttributeString", + "optionsSourceAssociationCaptionExpression", + "optionsSourceAssociationCustomContent", + "optionsSourceDatabaseCaptionExpression", + "optionsSourceDatabaseCustomContent", + "optionsSourceDatabaseValueAttribute", + } { + if !got[want] { + t.Errorf("dropped %s: a rule the single-condition vocabulary already lifted", want) + } + } +} + +// The other half of that distinction: a NEWLY supported guard shape in the same +// position IS withheld, because emitting it would ADD an over-firing rule rather +// than preserve an existing one. Datagrid hides pagingPosition only when +// pagination is off AND the row count is false, and pagination defaults on. +func TestConjunctionWithholdsANewShapeItCannotFullyRead(t *testing.T) { + js := `function getProperties(t,e){` + + `return e.pagination?x.hidePropertiesIn(t,e,["showNumberOfRows"]):` + + `(x.hidePropertiesIn(t,e,["showPagingButtons"]),!1===e.showNumberOfRows&&x.hidePropertiesIn(t,e,["pagingPosition"])),t}` + + rules, _ := extractVisibilityRulesFromJS(js) + for _, r := range rules { + if r.PropertyKey == "pagingPosition" && len(r.And) == 0 { + t.Errorf("emitted pagingPosition with only one conjunct (%s %s %q): "+ + "it is hidden when pagination is OFF *and* the row count is false, "+ + "and pagination defaults on", + r.HiddenWhen.PropertyKey, r.HiddenWhen.Operator, r.HiddenWhen.Value) + } + } + // Control: the sibling hide in the same group, whose guard the walk CAN + // attribute, is still lifted — so the test is not passing because extraction + // stopped altogether. + var sawSibling bool + for _, r := range rules { + if r.PropertyKey == "showPagingButtons" || r.PropertyKey == "showNumberOfRows" { + sawSibling = true + } + } + if !sawSibling { + t.Error("no rule survived at all; the withholding is too broad") + } +} From 45d8ff5fa184e826d944c46ec5b78330bb1f6c9c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 14:40:55 +0000 Subject: [PATCH 08/19] feat(check): resolve ALTER PAGE SET against the stored document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `alter page P { set NoSuchProperty = 10 on dgProducts; }` and `set PageSize = 12 on noSuchWidget;` both passed `mxcli check -p app.mpr --references` and were then refused by `exec`. Measured on a real 11.13.0 project. exec applies statements one at a time, so a script passed every pre-flight and stopped halfway, having written everything before the typo and nothing after — the inversion validate_alter_target.go closed for the ALTER's target document, one level further in. ValidateWidgetProperties resolves the properties of widgets a statement CARRIES: CREATE PAGE's tree, and ALTER's INSERT and REPLACE trees. A SET carries no widget. It names one already stored, whose vocabulary is partly a switch in setRawWidgetPropertyMut and partly the stored widget's own PropertyTypes — which belong to whatever widget package the project installed, so nothing in this repo can state it for an arbitrary project. So the check does not restate it. It opens the document, runs the real setter against a throwaway deep copy (Mutator.Probe, whose Save is refused) and keeps the error. Check and exec cannot drift, because there is one resolver; the author also gets exec's exact wording from the pre-flight, with the property keys the stored widget declares appended and a near-miss called out first. Two false-positive sources, both measured, both silence rather than a finding: a page the script itself creates (nothing stored — the document is not even opened), and a widget an INSERT in the same script adds. The second cannot be a name match — a DataGrid 2 column is inserted as `colBrand` and addressed as `Brand` — so the rule asks whether the target resolves. Gating on an optional interface assertion rather than on backend.PageMutator keeps the pass off the MCP mutator, which has no pluggable path and would have reported its own difference as the author's mistake. Also folds the pluggable setter's inline property-key derivation into buildPropKeyMap. It was a second copy of the same code, and it was #1069's bug site. Verification: - revert control: stubbing the pass makes the three gap tests fail - 4 casings of a real property pass (a case-sensitive pre-flight would have re-broken #1069); insert-then-set passes check AND exec in both the same-statement and across-statement forms - 17 ALTER SET examples in mdl-examples: 0 new errors vs a baseline binary built from main - the project is byte-identical (md5 over .mpr + mprcontents) after five check runs; probes refuse to save, with a control that real saves land Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .claude/skills/mendix/alter-page/SKILL.md | 15 +- .claude/skills/mendix/check-syntax/SKILL.md | 6 +- docs-wiki/bug-patterns/mutator-addressing.md | 16 + .../alter-page-set-property-unchecked.mdl | 89 ++++++ mdl/backend/pagemutator/mutator.go | 36 +-- .../pluggable_property_casing_test.go | 5 +- mdl/backend/pagemutator/probe.go | 105 +++++++ mdl/backend/pagemutator/probe_test.go | 164 ++++++++++ mdl/executor/cmd_alter_page.go | 69 +++-- mdl/executor/validate.go | 5 + mdl/executor/validate_alter_set.go | 248 +++++++++++++++ mdl/executor/validate_alter_set_test.go | 285 ++++++++++++++++++ 13 files changed, 986 insertions(+), 58 deletions(-) create mode 100644 mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl create mode 100644 mdl/backend/pagemutator/probe.go create mode 100644 mdl/backend/pagemutator/probe_test.go create mode 100644 mdl/executor/validate_alter_set.go create mode 100644 mdl/executor/validate_alter_set_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 2d00079fe8..5495427c1c 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -570,3 +570,4 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A module role written WITHOUT its module (`grant Wide on Sales.Order (read *)`) passes both `mxcli check` and `check --references`. The GRANT forms then fail at exec — after the preceding statements have been written — with the malformed message `failed to module not found for role .Wide`. `create user role R (Wide)` does not fail at all: it reports success and stores the reference as \".Wide\", and only MxBuild refuses the project, with CE1613 \"The selected module role '.Wide' no longer exists.\"", "cause": "The grammar spells a module role as `qualifiedName`, whose module part is optional (`identifierOrKeyword (DOT identifierOrKeyword)*`), so a bare name parses and reaches the executor with an empty Module. MDL-GRANT01 was the only check-time rule reading role lists, and it was written for the five DOCUMENT grants: `grant ... on `, the workflow grant and both user-role statements were never in its switch. On the document grants it did fire, but with the wrong diagnosis — it compared the empty module against the document's and reported a CROSS-MODULE error (CE0148), advice that does not fix a missing qualifier. The user-role path never validated at all; it concatenated `mr.Module + \".\" + mr.Name` and stored the result.", "file": "`mdl/executor/validate_grant_roles.go` (MDL-GRANT02, `moduleRoleList` + `validateRoleQualification`), `mdl/executor/cmd_security_write.go` (`qualifiedModuleRoleNames`, `validateModuleRole`)", "insight": "**Two switches over the same statement set, with nothing comparing them** — the same shape as the check-coverage defect fixed in `validate_duplicates.go` two days earlier (stmtCreateInfo 24 types vs setFor 20). Here it is the grammar's `moduleRoleList` (9 statements) against MDL-GRANT01's switch (5). When a rule reads a grammar list, enumerate the list, not the statements you happened to think of. **Order the two rules rather than merging them**: an unqualified role has no module to compare, so the cross-module check reads empty as \"some other module\" and produces a true-sounding message with the wrong remedy — qualification is settled first and suppresses the second rule for that statement. **A late error and a stored corruption are not the same severity.** The GRANT forms failed loudly at exec; the user-role form succeeded and wrote a dangling reference, which is worse and was found only by running the statement and then `mx check`. When auditing a validation gap, run each affected statement to the end rather than assuming they all fail the same way. Guard both layers: the pre-flight makes `exec` refuse before writing anything, and the executor guard still fires on the `-c` path, which skips the pre-flight — that is the control proving the fix is not only in the checker. Repros `mdl-examples/bug-tests/1067-unqualified-module-role.fail.mdl`. Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067", "#836"], "ce": ["CE1613", "CE0148"], "rules": ["MDL-GRANT02"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "After `alter entity E add attribute X`, the new attribute is invisible to some roles — it renders blank in the UI — while `check --references`, `mxbuild` and `mx check` all report the model clean (measured: 0 errors on 11.13.0). Reported as the attribute being \"added only to access rules with the widest member lists, and silently skipped on narrower rules\".", "cause": "Nothing is skipped. Measured on the raw BSON, the new attribute's qualified name occurs exactly ONCE PER RULE — every access rule gets its MemberAccess, which is why the model is structurally complete and builds clean. What differs is the RIGHTS: a new member joins each rule at that rule's `DefaultMemberAccessRights`, and MDL derives that property instead of setting it — `write *` gives ReadWrite, `read *` gives ReadOnly, and a grant written purely as member lists (`grant R on E (read (Subject))`) leaves it None. So the storage is right and matches what the property means; the defect was that `alter entity` printed only \"Added attribute 'X'\" and said nothing about the roles that had just been given None.", "file": "`mdl/executor/cmd_entities_new_member_access.go` (`rolesBlindToNewMembers`, `newMemberAccessWarning`), wired at the `ast.AlterEntityAddAttribute` branch of `mdl/executor/cmd_entities.go`", "insight": "**The reporter's mechanism was wrong and the symptom was right — take the symptom and re-derive the mechanism.** The control that settles it: a third rule granted `read *, write (Subject)` is NARROWER than `read *, write *` and still sees a newly added attribute, because `read *` set its default to ReadOnly. So the discriminator is the rule's default, not the width of its member list — and a fix aimed at \"copy the widest rule's members\" would have been wrong code for a real bug. **Count the BSON before believing a \"silently skipped\" report**: `grep -oa 'Mod.Ent.Attr' .mxunit | wc -l` distinguished \"entry missing\" (would be CE0066) from \"entry present at None\" (clean build) in one command, and they call for opposite fixes. A role named by ANY rule with a ReadOnly/ReadWrite default is not blind even if another of its rules is member-listed — Mendix combines the rights of every rule naming a role — so the report is a set difference, not a per-rule scan. Treat an EMPTY default as None: Mendix omits the property at its zero value, and reading empty as \"sees new members\" silences the warning on exactly the rules it exists for. Scope limit recorded in the code: the warning covers the entity's own rules, not a specialization's, because the descendants live in domain models the call does not hold. Repro `mdl-examples/bug-tests/1067-new-member-access-warning.mdl`. Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067", "#936"], "ce": ["CE0066"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A widget-describe change measured as 'zero rules lost' had in fact dropped SIX visibility rules from Combo box — the widget the work was justified by. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` (33 of them) but `mxcli widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42; the missing 9 included the one that mattered.", "cause": "The sweep's widget list was derived from an artifact of the pipeline (generated def.json files) rather than from the capability under test (what `widget describe` can describe). Nothing compared the two lists, so the sample silently excluded a whole class — the same 'two lists, nothing comparing them' shape as the defects being fixed.", "file": "mdl/executor/editorconfig_extract.go, mdl/executor/editorconfig_shapes_test.go", "insight": "Derive a sweep's population from the CAPABILITY, never from a convenient artifact, and state the population in the claim: 'zero rules lost across the 33 widgets carrying a def.json' would have been true and would have invited the question. A second trap immediately after: the regression test written to lock the fix used a hand-written editorConfig snippet of the same APPARENT shape, and it passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and the test proved nothing. Pointing the test at the real committed .mpk made the control fail with all six names. Rule of thumb: when a defect was found in real vendor input, the regression test takes the real input; a reconstructed minimal case must be shown to fail without the fix BEFORE it is trusted, and here it did not. The fix itself is the policy that should have been there from the start: conjunction support may WITHHOLD a rule the extractor never produced before (emitting one conjunct over-fires), but may never drop one the older vocabulary already lifted — that rule's accuracy is unchanged by the new work.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "`alter page P { set NoSuchProperty = 10 on dgProducts; }` and `set PageSize = 12 on noSuchWidget;` both passed `mxcli check -p app.mpr --references` (exit 0, \"All references valid\") and were then refused by `exec`: `pluggable property \"NoSuchProperty\" not found` / `widget \"noSuchWidget\" not found`. exec applies statements one at a time, so the script had already written everything before the typo", "cause": "ValidateWidgetProperties resolves the properties of widgets a statement CARRIES \u2014 CREATE PAGE's tree, and ALTER's InsertWidgetOp/ReplaceWidgetOp trees. SetPropertyOp carries no widget: it names one already stored, so its property can only be resolved against the DOCUMENT, which that pass never opens. Same inversion validate_alter_target.go closed for the ALTER's target document, one level further in", "file": "`mdl/executor/validate_alter_set.go` (new), `mdl/backend/pagemutator/probe.go` (new)", "insight": "**Do not re-derive what a mutation accepts \u2014 run the mutation.** The vocabulary of an ALTER SET is partly a switch in `setRawWidgetPropertyMut` and partly the STORED widget's own PropertyTypes, which belong to whatever widget package the project installed; no registry in this repo can state it for an arbitrary project. So the check opens the document, runs the real setter against a throwaway deep copy (`Mutator.Probe`, whose `Save` is refused), and keeps only the error \u2014 check and exec cannot drift because there is one resolver. The author also gets exec's exact wording from the pre-flight. **Two false-positive sources, both measured, both silence rather than a finding**: a page the script CREATEs (nothing stored \u2014 skip, and do not even open it) and a widget an INSERT in the same script adds. The second cannot be a name match: a DataGrid 2 column is inserted as `colBrand` and addressed as `Brand` (derived from the bound attribute), so the rule is `ResolvesTarget` \u2014 suppress only when the document does not carry the target AND the script adds widgets to that document. **Gating on an optional interface assertion, not on backend.PageMutator**, keeps the pass off the MCP mutator, which has no pluggable path and would have reported its own difference as the author's mistake. Controls: 4 casings of a real property pass (a case-sensitive pre-flight would have re-broken #1069); insert-then-set passes check AND exec in both the same-statement and across-statement forms; 17 ALTER SET examples in mdl-examples show 0 new errors vs the baseline binary; the project's files are byte-identical (md5 over .mpr + mprcontents) after five check runs. Revert control: stubbing the pass makes the three gap tests fail", "refs": []} diff --git a/.claude/skills/mendix/alter-page/SKILL.md b/.claude/skills/mendix/alter-page/SKILL.md index ad5ef9e182..bb6f783cfc 100644 --- a/.claude/skills/mendix/alter-page/SKILL.md +++ b/.claude/skills/mendix/alter-page/SKILL.md @@ -466,9 +466,16 @@ included, so a spelling `CREATE PAGE` accepts is a spelling `ALTER PAGE` accepts same statement. This is what makes DESCRIBE output re-executable: `describe page` prints the capitalised `PageSize:`, while the widget template stores `pageSize` (mendixlabs/mxcli#1069). A property the widget does not declare is still an -error, and it is the only signal you get — `mxcli check --references` does not -resolve pluggable property names, so a typo checks clean and fails at exec, after -earlier statements in the script have already been written. +error — and `mxcli check … --references` reports it **before** the script runs, +so a typo no longer lands halfway through. The pre-flight resolves the name +against the stored document rather than a list, so it is right about whatever +widget package this project has installed; the error names the widget's own +property keys. `ON` a widget the page does not have is caught the same way. + +Two things it deliberately stays quiet about, because it cannot answer them: a +page the script itself creates (nothing is stored yet — the widgets there are +checked where they are written), and a widget an `INSERT` in the same script +adds. Both still fail at exec if they are genuinely wrong. ## Common Mistakes @@ -477,7 +484,7 @@ earlier statements in the script have already been written. | Missing `on widgetName` for widget SET | Add `on widgetName` (only page-level properties — `Title`, `PopupWidth`, `PopupHeight`, `PopupResizable`, `Class`, `Style` — omit ON) | | `unsupported page-level property: title` | Page-level property names are case-sensitive — use `Title`, `PopupWidth`, `PopupHeight`, `PopupResizable`, `Class`, `Style` | | Using unquoted pluggable property names | Quote pluggable props: `set 'showLabel' = false on cb` | -| `pluggable property "X" not found` | The widget does not declare it — casing is not the problem (any casing resolves). Check the real name with `describe widget ` or `describe page` | +| `pluggable property "X" not found` | The widget does not declare it — casing is not the problem (any casing resolves). The error lists the keys it does declare; `describe widget ` or `describe page` shows them in context. Run `mxcli check … --references` to get this before the script runs | | Wrong widget name | Use `describe page Module.Name` to see widget names | | SET on non-existent widget | Widget names are case-sensitive; check with DESCRIBE | | Missing semicolons between operations | Each operation inside `{ }` ends with `;` | diff --git a/.claude/skills/mendix/check-syntax/SKILL.md b/.claude/skills/mendix/check-syntax/SKILL.md index 694e057bef..5a5909f65f 100644 --- a/.claude/skills/mendix/check-syntax/SKILL.md +++ b/.claude/skills/mendix/check-syntax/SKILL.md @@ -40,7 +40,11 @@ and mxcli's own rules; it does not validate the Mendix model. Run `mxcli check script.mdl` alone checks syntax and the semantic rules that need no model. **Pass `-p` and it also resolves every reference** — modules, entities, -pages, microflows and icons — against that project: +pages, microflows and icons — against that project. It reaches inside stored +documents where a name can only be answered there: an `ALTER PAGE … SET` is +dry-run against the page it edits, so a widget the page does not have, or a +property the stored widget does not declare, is reported here rather than +stopping the script partway through `exec`. ```bash mxcli check script.mdl # syntax + model-free rules diff --git a/docs-wiki/bug-patterns/mutator-addressing.md b/docs-wiki/bug-patterns/mutator-addressing.md index 33ff61b9a3..5434875439 100644 --- a/docs-wiki/bug-patterns/mutator-addressing.md +++ b/docs-wiki/bug-patterns/mutator-addressing.md @@ -66,6 +66,22 @@ single lookup scope holds two keys differing only in case, which across every shipped widget template is 0 of 1208 keys — and a test pins that as templates are added. +**The names a mutation may use are not a list, so the pre-flight runs the +mutation.** The other half of the same story is that `check` could not see any +of this: the properties reference checking resolved were the ones a statement +*carried*, and `SET` carries no widget — it names one already stored, whose +vocabulary is partly a switch in the setter and partly the installed widget +package's own template keys. Nothing in this repo can state that vocabulary for +an arbitrary project, so the check does not try: it opens the document, runs the +real setter against a throwaway copy, and keeps the error. Two resolvers that +must agree are cheaper to make one resolver than to keep in step — the drift +here is silent in the direction that hurts, a pre-flight that passes what the +run then refuses. The cost is that the copy has to be a real copy, which is one +test, and that a target the script itself adds has to be recognised as +not-yet-stored rather than missing — by asking whether it resolves, never by +matching names, since a grid column is inserted under one name and addressed +under its derived one. + **Hand-built BSON drifts from codec-built BSON.** The mutator constructs documents directly while CREATE goes through the codec, so the two encodings of "the same" widget diverge — an empty-string value where the codec writes an diff --git a/mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl b/mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl new file mode 100644 index 0000000000..886ecd1de6 --- /dev/null +++ b/mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl @@ -0,0 +1,89 @@ +-- ============================================================================ +-- `ALTER PAGE ... SET ON ` was not resolved by `mxcli check` +-- ============================================================================ +-- +-- Symptom (before fix): +-- Both of these passed `mxcli check -p app.mpr --references` — exit 0, "All +-- references valid" — and were then refused by `exec`: +-- +-- set NoSuchProperty = 10 on dgProducts; +-- Error: failed to set: failed to set NoSuchProperty on dgProducts: +-- pluggable property "NoSuchProperty" not found +-- set PageSize = 12 on noSuchWidget; +-- Error: failed to set: failed to set PageSize on noSuchWidget: +-- widget "noSuchWidget" not found +-- +-- The inversion validate_alter_target.go describes, one level further in: +-- check is meant to be the strict gate and exec the thing that runs, so a +-- script passed every pre-flight and then stopped halfway, having applied the +-- statements before the typo and none after. +-- +-- Root cause: +-- ValidateWidgetProperties resolves the properties of widgets a statement +-- CARRIES — CREATE PAGE's tree, and ALTER's INSERT and REPLACE trees. A SET +-- carries no widget: it names one that is already stored, so its property can +-- only be resolved against the DOCUMENT, which that pass never opens. +-- +-- After fix: +-- `check --references` dry-runs each SET against a throwaway copy of the +-- stored document (pagemutator.Probe) and reports the setter's own error, +-- with the property keys the stored widget declares appended. The vocabulary +-- is not restated anywhere: a pluggable widget's keys belong to whatever +-- widget package the project has installed, so the only way to be right about +-- them is to ask the document. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl -p app.mpr +-- mxcli check mdl-examples/bug-tests/alter-page-set-property-unchecked.mdl -p app.mpr --references +-- +-- This file is the WORKING half: every statement below must pass check and +-- exec. The two failing spellings are in the header above, on purpose — a +-- bug-test that always exits 1 cannot be run in the suite. +-- ============================================================================ + +create entity MyFirstModule.Product ( Title: String(200), Brand: String(100) ); + +create or replace page MyFirstModule.P_SetChecked +( + Title: 'Set checked', + Layout: Atlas_Core.Atlas_Default +) +{ + container cGrid { + datagrid dgProducts (DataSource: database MyFirstModule.Product, PageSize: 20) { + column colTitle (Attribute: Title, Caption: 'Title') + } + } +} + +-- A property the widget really declares, in each of the spellings an author +-- writes: the one DESCRIBE PAGE prints, the one the template stores, and the +-- flat cases. A check that resolved names case-sensitively would refuse the +-- tool's own DESCRIBE output — that was mendixlabs/mxcli#1069, and this is the +-- control that the pre-flight did not reintroduce it. +alter page MyFirstModule.P_SetChecked { + set PageSize = 12 on dgProducts; + set pageSize = 13 on dgProducts; + set pagesize = 14 on dgProducts; +} + +-- A first-class property on a built-in widget: resolved by the setter's switch, +-- not by anything the document declares, so the check must let it through in +-- any casing (mendixlabs/mxcli#1069's sibling — see +-- alter-page-lowercase-set-on-builtin.mdl). +alter page MyFirstModule.P_SetChecked { + set class = 'fl-grid-wrap' on cGrid; + set Class = 'fl-grid-wrap mx-spacing-top-large' on cGrid; +} + +-- A widget the script itself adds is not in the stored document, and must not +-- be reported as missing. Only its EXISTENCE is affected — a target that DOES +-- resolve is still checked. Note the column is addressed by its DERIVED name +-- (`Brand`, the bound attribute), not the `colBrand` written in the INSERT, +-- which is why the suppression cannot be a name match. +alter page MyFirstModule.P_SetChecked { + insert after dgProducts.Title { column colBrand (Attribute: Brand, Caption: 'Brand') } + set Caption = 'Brand name' on dgProducts.Brand; +} + +describe page MyFirstModule.P_SetChecked; diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index c17074ef7e..7c61e94be3 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -54,6 +54,9 @@ type Mutator struct { unitID model.ID deps Deps widgetFinder widgetFinder + // probe marks a discardable copy handed out by Probe(), which exists to be + // written to and thrown away. Save refuses on one — see probe.go. + probe bool } // New constructs a Mutator over an already-decoded unit document. It derives the @@ -1072,6 +1075,9 @@ func (m *Mutator) FindWidget(name string) bool { } func (m *Mutator) Save() error { + if m.probe { + return fmt.Errorf("refusing to save a dry-run copy of this %s", m.containerType) + } outBytes, err := bson.Marshal(m.rawData) if err != nil { return fmt.Errorf("marshal modified %s: %w", m.containerType, err) @@ -2798,34 +2804,20 @@ func setWidgetAttributeRefMut(widget bson.D, value any) error { // PropertyTypes, and no shipped template or definition has two keys in the same // list differing only in case (96 scopes, 1208 keys, 0 collisions — held by // TestPluggablePropertyKeysAreUniqueIgnoringCase). An unknown property still -// errors, which is the author's only signal: `check --references` does not -// resolve pluggable property names. +// errors — and `mxcli check --references` now reaches that error before the +// script runs, by dry-running this setter against a copy of the document rather +// than re-deriving what it accepts (see probe.go). func setPluggableWidgetPropertyMut(widget bson.D, propName string, value any) error { obj := bsonnav.DGetDoc(widget, "Object") if obj == nil { return fmt.Errorf("property %q not found (widget has no pluggable Object)", propName) } - propTypeKeyMap := make(map[string]string) - if widgetType := bsonnav.DGetDoc(widget, "Type"); widgetType != nil { - if objType := bsonnav.DGetDoc(widgetType, "ObjectType"); objType != nil { - propTypes := bsonnav.DGetArrayElements(bsonnav.DGet(objType, "PropertyTypes")) - for _, pt := range propTypes { - ptDoc, ok := pt.(bson.D) - if !ok { - continue - } - key := bsonnav.DGetString(ptDoc, "PropertyKey") - if key == "" { - continue - } - id := bsonnav.ExtractBinaryIDFromDoc(bsonnav.DGet(ptDoc, "$ID")) - if id != "" { - propTypeKeyMap[id] = key - } - } - } - } + // The same derivation buildPropKeyMap does, and it used to be spelled out a + // second time here. One of the two copies was #1069's bug site, so they are + // now one function — a resolver that disagrees with itself is the failure + // this whole area keeps producing. + propTypeKeyMap := buildPropKeyMap(widget) props := bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) for _, prop := range props { diff --git a/mdl/backend/pagemutator/pluggable_property_casing_test.go b/mdl/backend/pagemutator/pluggable_property_casing_test.go index f3851d6766..d8fdba84d0 100644 --- a/mdl/backend/pagemutator/pluggable_property_casing_test.go +++ b/mdl/backend/pagemutator/pluggable_property_casing_test.go @@ -107,8 +107,9 @@ func TestSetPluggableProperty_MatchesTemplateKeyRegardlessOfCase(t *testing.T) { // TestSetPluggableProperty_UnknownPropertyStillErrors is the control for the // test above: relaxing the comparison to case-insensitive must not turn a // genuinely unknown property into a silent no-op. A typo has to keep failing — -// that is the only signal the author gets, since `mxcli check --references` -// does not resolve pluggable property names. +// and this error is now also what `mxcli check --references` reports, because +// the pre-flight dry-runs this setter rather than re-deriving what it accepts +// (probe.go). Weakening it here would go quiet in two places at once. func TestSetPluggableProperty_UnknownPropertyStillErrors(t *testing.T) { rawData := makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")) m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} diff --git a/mdl/backend/pagemutator/probe.go b/mdl/backend/pagemutator/probe.go new file mode 100644 index 0000000000..8efb8e2240 --- /dev/null +++ b/mdl/backend/pagemutator/probe.go @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "fmt" + "sort" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// A dry run, so `mxcli check` can refuse what `exec` refuses. +// +// The property names an ALTER PAGE `SET` accepts are not a list that exists +// anywhere. A first-class name is whatever setRawWidgetPropertyMut's switch +// happens to handle; a pluggable one is whatever the STORED widget's own +// PropertyTypes declare, which is specific to the widget package the project has +// installed — no registry, and no table in this repo, can answer it for an +// arbitrary project. +// +// So the check does not re-derive the rule. It runs the real setter against a +// copy of the document and keeps only the error. That matters more than the code +// it saves: a check that re-implemented the resolution would drift from the +// setter, and the drift is silent in exactly the direction that hurts — a check +// that passes what exec then refuses, which is the gap this closes. The copy's +// Save is refused, so a mistake in a caller cannot turn a check into a write. +// +// Probe returns a copy of this mutator over its own deep copy of the document. +// Writes to it are visible nowhere: not in this mutator, and not on disk. +func (m *Mutator) Probe() (backend.PageMutator, error) { + raw, err := bson.Marshal(m.rawData) + if err != nil { + return nil, fmt.Errorf("probe %s: marshal: %w", m.containerType, err) + } + var cloned bson.D + if err := bson.Unmarshal(raw, &cloned); err != nil { + return nil, fmt.Errorf("probe %s: unmarshal: %w", m.containerType, err) + } + return &Mutator{ + rawData: cloned, + containerType: m.containerType, + unitID: m.unitID, + deps: m.deps, + widgetFinder: m.widgetFinder, + probe: true, + }, nil +} + +// ResolvesTarget reports whether the stored document carries what this +// reference names — a widget, or a grid column when columnRef is set. It +// answers the question the setters answer first, so a caller can tell a target +// that is missing from one whose property is wrong, without reading an error +// message to find out which. +func (m *Mutator) ResolvesTarget(widgetRef, columnRef string) bool { + if widgetRef == "" { + return true // page-level SET addresses the document itself + } + if columnRef != "" { + _, err := findBsonColumn(m.rawData, widgetRef, columnRef, m.widgetFinder) + return err == nil + } + return m.widgetFinder(m.rawData, widgetRef) != nil +} + +// WidgetPropertyKeys returns the property names the STORED widget declares — a +// pluggable widget's own template keys, or a DataGrid 2 column's when columnRef +// names one. It is the vocabulary a failed `SET` should be measured against, so +// the author sees what the widget has rather than only that their spelling is +// not it. +// +// A built-in widget returns nothing: its vocabulary is the setter's switch, not +// anything the document carries, and reporting an empty list as "this widget has +// no properties" would be worse than saying nothing. +func (m *Mutator) WidgetPropertyKeys(widgetRef, columnRef string) []string { + byID := map[string]string{} + if columnRef != "" { + result, err := findBsonColumn(m.rawData, widgetRef, columnRef, m.widgetFinder) + if err != nil || result == nil { + return nil + } + byID = result.colPropKeys + } else { + result := m.widgetFinder(m.rawData, widgetRef) + if result == nil { + return nil + } + byID = result.colPropKeys + if len(byID) == 0 { + byID = buildPropKeyMap(result.widget) + } + } + seen := make(map[string]bool, len(byID)) + keys := make([]string, 0, len(byID)) + for _, key := range byID { + if key == "" || seen[key] { + continue + } + seen[key] = true + keys = append(keys, key) + } + sort.Strings(keys) + return keys +} diff --git a/mdl/backend/pagemutator/probe_test.go b/mdl/backend/pagemutator/probe_test.go new file mode 100644 index 0000000000..064b421bfa --- /dev/null +++ b/mdl/backend/pagemutator/probe_test.go @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// recordingDeps counts the one thing a dry run must never do. +type recordingDeps struct{ saves int } + +func (d *recordingDeps) SerializeWidget(pages.Widget) bson.D { return nil } +func (d *recordingDeps) SerializeClientAction(pages.ClientAction) bson.D { return nil } +func (d *recordingDeps) SerializeCustomWidgetDataSource(pages.DataSource) bson.D { + return nil +} +func (d *recordingDeps) BuildDataGrid2Column(*backend.DataGridColumnSpec, string, map[string]pages.PropertyTypeIDEntry) (bson.D, error) { + return nil, nil +} +func (d *recordingDeps) SaveUnit(string, []byte) error { d.saves++; return nil } + +// TestProbeWritesReachNothing is the property the whole check-time dry run rests +// on: a probe is written to, and neither the mutator it came from nor storage +// sees it. +// +// The second half is the control. Without it the test passes against a Probe +// that returns a mutator over a document nothing can change — which would also +// leave the original alone, and would report every property as fine. +func TestProbeWritesReachNothing(t *testing.T) { + deps := &recordingDeps{} + raw := makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")) + m := New(raw, "unit-1", deps) + + probe, err := m.Probe() + if err != nil { + t.Fatalf("probe: %v", err) + } + if err := probe.SetWidgetProperty("dgProducts", "pageSize", 10); err != nil { + t.Fatalf("set on probe: %v", err) + } + if got := pluggablePrimitive(t, raw, "dgProducts"); got != "20" { + t.Errorf("original pageSize = %q after a probe write, want it untouched at %q", got, "20") + } + if deps.saves != 0 { + t.Errorf("SaveUnit called %d times during a dry run, want 0", deps.saves) + } + + // Control: the same write, on the mutator itself, does land. Without this + // the assertion above is satisfied by a probe that writes nowhere at all. + if err := m.SetWidgetProperty("dgProducts", "pageSize", 10); err != nil { + t.Fatalf("set on mutator: %v", err) + } + if got := pluggablePrimitive(t, raw, "dgProducts"); got != "10" { + t.Errorf("pageSize = %q after a real write, want %q", got, "10") + } +} + +// TestProbeRefusesToSave guards the direction that would turn `mxcli check` into +// a write: a caller that dry-runs an operation and then, by mistake, persists +// the copy. +func TestProbeRefusesToSave(t *testing.T) { + deps := &recordingDeps{} + m := New(makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")), "unit-1", deps) + + probe, err := m.Probe() + if err != nil { + t.Fatalf("probe: %v", err) + } + if err := probe.Save(); err == nil { + t.Fatal("Save on a probe returned nil, want a refusal") + } + if deps.saves != 0 { + t.Errorf("SaveUnit called %d times, want 0", deps.saves) + } + // Control: the mutator it came from still saves. + if err := m.Save(); err != nil { + t.Fatalf("Save on the real mutator: %v", err) + } + if deps.saves != 1 { + t.Errorf("SaveUnit called %d times on a real save, want 1", deps.saves) + } +} + +// TestProbeReportsWhatTheSetterWouldReport pins the point of the dry run: the +// error a probe produces is the setter's own, so an author reads the same +// sentence from `check` that `exec` would have given them. +func TestProbeReportsWhatTheSetterWouldReport(t *testing.T) { + m := New(makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")), "unit-1", &recordingDeps{}) + probe, err := m.Probe() + if err != nil { + t.Fatalf("probe: %v", err) + } + + probeErr := probe.SetWidgetProperty("dgProducts", "NoSuchProperty", 10) + if probeErr == nil { + t.Fatal("probe accepted an unknown pluggable property") + } + realErr := m.SetWidgetProperty("dgProducts", "NoSuchProperty", 10) + if realErr == nil { + t.Fatal("the setter accepted an unknown pluggable property") + } + if probeErr.Error() != realErr.Error() { + t.Errorf("probe error %q != setter error %q", probeErr, realErr) + } +} + +// TestWidgetPropertyKeysNamesTheStoredTemplateKeys — the hint the check appends +// comes from the document, not from a widget definition on disk, so it is right +// for whatever widget package this project happens to have installed. +func TestWidgetPropertyKeysNamesTheStoredTemplateKeys(t *testing.T) { + m := New(makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")), "unit-1", &recordingDeps{}) + + keys := m.WidgetPropertyKeys("dgProducts", "") + if len(keys) != 1 || keys[0] != "pageSize" { + t.Errorf("WidgetPropertyKeys = %v, want [pageSize]", keys) + } + if got := m.WidgetPropertyKeys("noSuchWidget", ""); got != nil { + t.Errorf("WidgetPropertyKeys for a missing widget = %v, want nil", got) + } + // A built-in widget declares nothing in the document; its vocabulary is the + // setter's switch. Reporting an empty list as the widget's properties would + // be worse than saying nothing, so the caller must get nothing. + m2 := New(makeRawPage(makeWidget("topBar", "Forms$DivContainer")), "unit-2", &recordingDeps{}) + if got := m2.WidgetPropertyKeys("topBar", ""); len(got) != 0 { + t.Errorf("WidgetPropertyKeys for a built-in widget = %v, want none", got) + } +} + +// TestResolvesTargetSeparatesMissingFromWrong lets a caller tell a target that +// is not there from one whose property is wrong, without parsing an error +// message to find out which. +func TestResolvesTargetSeparatesMissingFromWrong(t *testing.T) { + m := New(makeRawPage(makePluggableWidget("dgProducts", "pageSize", "20")), "unit-1", &recordingDeps{}) + + if !m.ResolvesTarget("dgProducts", "") { + t.Error("stored widget did not resolve") + } + if m.ResolvesTarget("noSuchWidget", "") { + t.Error("missing widget resolved") + } + if !m.ResolvesTarget("", "") { + t.Error("page-level target did not resolve") + } + if m.ResolvesTarget("dgProducts", "NoSuchColumn") { + t.Error("missing column resolved") + } +} + +func TestProbeErrorMentionsTheContainerKind(t *testing.T) { + m := New(makeRawPage(), "unit-1", &recordingDeps{}) + probe, err := m.Probe() + if err != nil { + t.Fatalf("probe: %v", err) + } + if err := probe.Save(); err == nil || !strings.Contains(err.Error(), "page") { + t.Errorf("Save refusal = %v, want it to name the container kind", err) + } +} diff --git a/mdl/executor/cmd_alter_page.go b/mdl/executor/cmd_alter_page.go index c3c8b31677..b856778f66 100644 --- a/mdl/executor/cmd_alter_page.go +++ b/mdl/executor/cmd_alter_page.go @@ -29,46 +29,22 @@ func execAlterPage(ctx *ExecContext, s *ast.AlterPageStmt) error { return mdlerrors.NewBackend("build hierarchy", err) } - var unitID model.ID - var containerID model.ID - containerType := strings.ToLower(s.ContainerType) - if containerType == "" { - containerType = "page" + unitID, containerID, containerType, err := resolveAlterPageUnit(ctx, s, h) + if err != nil { + return err } - switch containerType { - case "snippet": - snippet, modID, err := findSnippetByName(ctx, s.PageName, h) - if err != nil { - return err - } - unitID = snippet.ID - containerID = modID - case "layout": - layout, err := findLayoutByQName(ctx, s.PageName) - if err != nil { - return err - } - modID := h.FindModuleID(layout.ContainerID) + if containerType == "layout" { // The same refusal CREATE LAYOUT makes, for the same reason: a // Marketplace update replaces the module wholesale, so an edit here is // gone at the next update with nothing to show it ever happened. - if mod, _ := ctx.Backend.GetModule(modID); isMarketplaceModule(ctx, mod) { + if mod, _ := ctx.Backend.GetModule(containerID); isMarketplaceModule(ctx, mod) { return mdlerrors.NewValidation(fmt.Sprintf( "layout %s is in a marketplace module — an edit there is overwritten by the next module update. "+ "Copy it into a module of your own first: `describe layout %s`, rename it, run it, "+ "then repoint pages with `alter pages set layout = where layout = %s`", s.PageName.String(), s.PageName.String(), s.PageName.String())) } - unitID = layout.ID - containerID = modID - default: - page, err := findPageByName(ctx, s.PageName, h) - if err != nil { - return err - } - unitID = page.ID - containerID = h.FindModuleID(page.ContainerID) } // Open the page for mutation via the backend @@ -135,6 +111,41 @@ func execAlterPage(ctx *ExecContext, s *ast.AlterPageStmt) error { return nil } +// resolveAlterPageUnit resolves an ALTER PAGE / SNIPPET / LAYOUT target to the +// storage unit it edits and the module holding it. One statement type, three +// document kinds — the visitor sets ContainerType from the keyword, and an empty +// one means PAGE. +// +// It is shared with the check-time dry run (validate_alter_set.go) so both +// address the same document: a pre-flight that resolved the target differently +// from exec would be checking a different page than the one about to change. +func resolveAlterPageUnit(ctx *ExecContext, s *ast.AlterPageStmt, h *ContainerHierarchy) (unitID, containerID model.ID, containerType string, err error) { + containerType = strings.ToLower(s.ContainerType) + if containerType == "" { + containerType = "page" + } + switch containerType { + case "snippet": + snippet, modID, err := findSnippetByName(ctx, s.PageName, h) + if err != nil { + return "", "", containerType, err + } + return snippet.ID, modID, containerType, nil + case "layout": + layout, err := findLayoutByQName(ctx, s.PageName) + if err != nil { + return "", "", containerType, err + } + return layout.ID, h.FindModuleID(layout.ContainerID), containerType, nil + default: + page, err := findPageByName(ctx, s.PageName, h) + if err != nil { + return "", "", containerType, err + } + return page.ID, h.FindModuleID(page.ContainerID), containerType, nil + } +} + // ============================================================================ // SET property via mutator // ============================================================================ diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 74153e1e91..fc507b3689 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -316,6 +316,11 @@ func validateProgram(ctx *ExecContext, prog *ast.Program) []error { // member that does not exist reached mxbuild as CE1613 // (mendixlabs/mxcli#1049). errors = append(errors, validateXPathMembers(ctx, prog)...) + // Dry-run every ALTER … SET against the stored document. The properties of a + // widget the statement CARRIES are checked without a project; a SET names a + // widget that is already stored, so its property can only be resolved + // against the document — which is why it passed check and failed exec. + errors = append(errors, validateAlterSetProperties(ctx, prog, sc)...) return errors } diff --git a/mdl/executor/validate_alter_set.go b/mdl/executor/validate_alter_set.go new file mode 100644 index 0000000000..6b04aae032 --- /dev/null +++ b/mdl/executor/validate_alter_set.go @@ -0,0 +1,248 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// `ALTER PAGE … SET ON ` was the one widget-property position +// reference checking never looked at. +// +// ValidateWidgetProperties resolves the properties of widgets a statement +// CARRIES — the trees under CREATE PAGE, and under ALTER's INSERT and REPLACE — +// against the widget definitions. A `SET` carries no widget: it names one that +// is already stored, so its property has to be resolved against the DOCUMENT, +// which that pass never opens. Both of these passed `check -p --references` and +// were then refused by `exec`, on a real 11.13.0 project: +// +// set NoSuchProperty = 10 on dgProducts; +// exec -> pluggable property "NoSuchProperty" not found +// set PageSize = 12 on noSuchWidget; +// exec -> widget "noSuchWidget" not found +// +// Same inversion validate_alter_target.go describes for the document itself: +// check is meant to be the strict gate and exec the thing that runs, and here +// exec was stricter — so a script passed every pre-flight and then stopped +// halfway, having applied the statements before the typo and none after. +// +// What makes this checkable at all is that the answer is not re-derived. The +// vocabulary a `SET` accepts is partly a switch in the setter and partly the +// stored widget's own template keys, which belong to whatever widget package the +// project has installed; anything that restated it here would drift, and drift +// silently in the direction that hurts. Instead the setter is RUN, against a +// throwaway copy of the document (pagemutator.Probe), and only its error is +// kept. Check and exec therefore disagree only if the copy differs from the +// original, which is the one thing that is cheap to guarantee. + +// pageProbe is the optional capability the shared BSON page mutator offers: a +// discardable copy to dry-run an operation against, and the property names the +// stored widget declares. +// +// It is an assertion rather than an addition to backend.PageMutator on purpose. +// The MCP mutator has neither — and no pluggable path at all — so asserting is +// also what keeps this pass off a backend whose SET support is different, rather +// than reporting that difference as the author's mistake. +type pageProbe interface { + Probe() (backend.PageMutator, error) + ResolvesTarget(widgetRef, columnRef string) bool + WidgetPropertyKeys(widgetRef, columnRef string) []string +} + +// validateAlterSetProperties dry-runs every ALTER … SET against the stored +// document and reports what exec would refuse. +func validateAlterSetProperties(ctx *ExecContext, prog *ast.Program, sc *scriptContext) []error { + if prog == nil || !ctx.Connected() { + return nil + } + h, err := getHierarchy(ctx) + if err != nil || h == nil { + return nil + } + grows := documentsTheScriptAddsWidgetsTo(prog) + opened := map[model.ID]pageProbe{} + + var errs []error + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.AlterPageStmt) + if !ok || !hasSetPropertyOp(s) || alterTargetComesFromScript(sc, s) { + continue + } + unitID, containerID, containerType, err := resolveAlterPageUnit(ctx, s, h) + if err != nil { + // A target that does not resolve is validateAlterTarget's finding, + // not this one's. Reporting it twice, in two wordings, reads as two + // problems. + continue + } + probe, ok := opened[unitID] + if !ok { + probe = openPageProbe(ctx, unitID) + opened[unitID] = probe + } + if probe == nil { + continue + } + label := fmt.Sprintf("alter %s %s", containerType, s.PageName.String()) + modName := h.GetModuleName(containerID) + for _, op := range s.Operations { + set, ok := op.(*ast.SetPropertyOp) + if !ok { + continue + } + // A widget an INSERT or REPLACE in this script puts on the page is + // not in the stored document, so a dry run would report it missing. + // Only its EXISTENCE is affected, though — a target that does + // resolve is checked as normal — and matching by name would not + // work anyway: a DataGrid 2 column is addressed by a derived name, + // so `column colTitle` is inserted under one name and set under + // another. Its properties are checked where it is written, by + // ValidateWidgetProperties. + if grows[s.PageName.String()] && !probe.ResolvesTarget(set.Target.Widget, set.Target.Column) { + continue + } + errs = append(errs, checkSetOp(ctx, probe, label, set, modName, containerID)...) + } + } + return errs +} + +// openPageProbe opens the stored document and returns it as a prober, or nil +// when it cannot be established — an engine without the capability, a unit that +// will not load. Silence, never a finding: a false "no such property" blocks a +// script that would have worked, which is worse than the gap it replaces. +func openPageProbe(ctx *ExecContext, unitID model.ID) pageProbe { + mutator, err := ctx.Backend.OpenPageForMutation(unitID) + if err != nil { + return nil + } + probe, ok := mutator.(pageProbe) + if !ok { + return nil + } + return probe +} + +// checkSetOp dry-runs one SET, one property at a time. +// +// Per property rather than per op, on its own copy each time, so a statement +// setting four properties reports all four mistakes instead of stopping at the +// first — the difference between one round of correction and four. +func checkSetOp(ctx *ExecContext, p pageProbe, label string, op *ast.SetPropertyOp, modName string, modID model.ID) []error { + names := make([]string, 0, len(op.Properties)) + for name := range op.Properties { + names = append(names, name) + } + sort.Strings(names) + + var errs []error + for _, name := range names { + probe, err := p.Probe() + if err != nil { + return errs + } + one := &ast.SetPropertyOp{ + Target: op.Target, + Properties: map[string]any{name: op.Properties[name]}, + } + if err := applySetPropertyMutator(ctx, probe, one, modName, modID); err != nil { + errs = append(errs, mdlerrors.NewValidation(fmt.Sprintf( + "%s: %v%s", label, err, declaredPropertyHint(p, op.Target, name)))) + } + } + return errs +} + +// declaredPropertyHint names what the widget does have, when the widget resolves +// and declares its own property keys. A near-miss is called out first: the +// spelling of a pluggable key is the thing authors get wrong (it is lowerCamel +// in the template, capitalised in DESCRIBE output), so `pagesize` against +// `pageSize` should read as a typo and not as a list to search. +func declaredPropertyHint(p pageProbe, target ast.WidgetRef, prop string) string { + if target.Widget == "" { + return "" // page-level SET: the setter's own error lists what it takes + } + keys := p.WidgetPropertyKeys(target.Widget, target.Column) + if len(keys) == 0 { + return "" + } + for _, k := range keys { + if strings.EqualFold(k, prop) { + return "" // it does have it — the failure is about the value + } + } + hint := "" + if near := nearestKey(prop, keys); near != "" { + hint = fmt.Sprintf(" — did you mean `%s`?", near) + } + const max = 8 + if len(keys) > max { + return fmt.Sprintf("%s (%s declares %s and %d more)", + hint, target.Name(), strings.Join(keys[:max], ", "), len(keys)-max) + } + return fmt.Sprintf("%s (%s declares %s)", hint, target.Name(), strings.Join(keys, ", ")) +} + +// hasSetPropertyOp reports whether a statement carries anything this pass would +// look at, so a script of pure INSERTs never opens a document. +func hasSetPropertyOp(s *ast.AlterPageStmt) bool { + for _, op := range s.Operations { + if _, ok := op.(*ast.SetPropertyOp); ok { + return true + } + } + return false +} + +// alterTargetComesFromScript reports whether the document this ALTER edits is +// one the script itself creates — in which case there is nothing stored to dry +// run against, and the widgets it names are checked where they are written. +func alterTargetComesFromScript(sc *scriptContext, s *ast.AlterPageStmt) bool { + if sc == nil { + return false + } + qn := s.PageName.String() + if s.PageName.Module == "" || sc.modules[s.PageName.Module] { + return true + } + switch strings.ToUpper(s.ContainerType) { + case "SNIPPET": + return sc.snippets[qn] + case "LAYOUT": + return sc.layouts[qn] + default: + return sc.pages[qn] + } +} + +// documentsTheScriptAddsWidgetsTo names the documents an INSERT or REPLACE +// somewhere in the script puts new widgets on — the ones whose stored widget +// tree is not the tree a later SET will run against. +// +// Deliberately not ordered: a SET before the INSERT that adds its target fails +// at exec too, so suppressing it here costs one unreported error, while getting +// the order wrong the other way costs a false one on a script that works. Those +// are not comparable — the second blocks the run. +func documentsTheScriptAddsWidgetsTo(prog *ast.Program) map[string]bool { + grows := map[string]bool{} + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.AlterPageStmt) + if !ok { + continue + } + for _, op := range s.Operations { + switch op.(type) { + case *ast.InsertWidgetOp, *ast.ReplaceWidgetOp: + grows[s.PageName.String()] = true + } + } + } + return grows +} diff --git a/mdl/executor/validate_alter_set_test.go b/mdl/executor/validate_alter_set_test.go new file mode 100644 index 0000000000..106d0b461e --- /dev/null +++ b/mdl/executor/validate_alter_set_test.go @@ -0,0 +1,285 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/backend/pagemutator" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// --------------------------------------------------------------------------- +// A stored page carrying one pluggable widget +// --------------------------------------------------------------------------- + +// storedGridPage builds the raw document of a page holding a pluggable widget +// named dgProducts with the template key `pageSize` — the shape #1069 was about, +// and the one a `SET` has to be resolved against, since no widget definition on +// disk can say what THIS project's installed grid declares. +func storedGridPage() bson.D { + typeID := primitive.Binary{Subtype: 0x04, Data: []byte{ + 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, + 0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff, 0x00, + }} + grid := bson.D{ + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Name", Value: "dgProducts"}, + {Key: "Type", Value: bson.D{ + {Key: "$Type", Value: "CustomWidgets$CustomWidgetType"}, + {Key: "ObjectType", Value: bson.D{ + {Key: "PropertyTypes", Value: bson.A{ + int32(2), + bson.D{ + {Key: "$ID", Value: typeID}, + {Key: "PropertyKey", Value: "pageSize"}, + }, + }}, + }}, + }}, + {Key: "Object", Value: bson.D{ + {Key: "Properties", Value: bson.A{ + int32(2), + bson.D{ + {Key: "TypePointer", Value: typeID}, + {Key: "Value", Value: bson.D{{Key: "PrimitiveValue", Value: "20"}}}, + }, + }}, + }}, + } + return bson.D{ + {Key: "$Type", Value: "Forms$Page"}, + {Key: "FormCall", Value: bson.D{ + {Key: "Arguments", Value: bson.A{ + int32(2), + bson.D{{Key: "Widgets", Value: bson.A{int32(2), grid}}}, + }}, + }}, + } +} + +// countingDeps records the writes a validation pass must not make. +type countingDeps struct{ saves int } + +func (d *countingDeps) SerializeWidget(pages.Widget) bson.D { return nil } +func (d *countingDeps) SerializeClientAction(pages.ClientAction) bson.D { return nil } +func (d *countingDeps) SerializeCustomWidgetDataSource(pages.DataSource) bson.D { + return nil +} +func (d *countingDeps) BuildDataGrid2Column(*backend.DataGridColumnSpec, string, map[string]pages.PropertyTypeIDEntry) (bson.D, error) { + return nil, nil +} +func (d *countingDeps) SaveUnit(string, []byte) error { d.saves++; return nil } + +// gridPageCtx wires a project holding exactly one page, MyModule.P_Grid, whose +// stored document is storedGridPage(). +func gridPageCtx(t *testing.T) (*ExecContext, *countingDeps, *int) { + t.Helper() + mod := mkModule("MyModule") + pg := mkPage(mod.ID, "P_Grid") + deps := &countingDeps{} + opens := 0 + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListFoldersFunc: func() ([]*types.FolderInfo, error) { return nil, nil }, + ListPagesFunc: func() ([]*pages.Page, error) { return []*pages.Page{pg}, nil }, + OpenPageForMutationFunc: func(unitID model.ID) (backend.PageMutator, error) { + opens++ + return pagemutator.New(storedGridPage(), unitID, deps), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + return ctx, deps, &opens +} + +// parseMDL builds a program from real MDL, so the test exercises the AST the +// visitor actually produces rather than one written to suit the validator. +func parseMDL(t *testing.T, src string) *ast.Program { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse %q: %v", src, errs) + } + if prog == nil { + t.Fatalf("parse %q: nil program", src) + } + return prog +} + +func checkAlterSet(t *testing.T, ctx *ExecContext, src string) []error { + t.Helper() + prog := parseMDL(t, src) + sc := newScriptContext() + sc.collectDefinitions(prog) + return validateAlterSetProperties(ctx, prog, sc) +} + +// --------------------------------------------------------------------------- +// The gap +// --------------------------------------------------------------------------- + +// TestAlterSet_UnknownPluggableProperty is the regression test for the reported +// gap: measured on a real 11.13.0 project, `set NoSuchProperty = 10 on +// dgProducts` passed `check -p --references` and was then refused by `exec` with +// `pluggable property "NoSuchProperty" not found`. +func TestAlterSet_UnknownPluggableProperty(t *testing.T) { + ctx, deps, _ := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `alter page MyModule.P_Grid { set NoSuchProperty = 10 on dgProducts; }`) + if len(errs) != 1 { + t.Fatalf("got %d errors, want 1: %v", len(errs), errs) + } + msg := errs[0].Error() + for _, want := range []string{"MyModule.P_Grid", "NoSuchProperty", "dgProducts"} { + if !strings.Contains(msg, want) { + t.Errorf("error %q does not name %q", msg, want) + } + } + // The hint names what the STORED widget declares, so the author is not left + // guessing a spelling — the mistake #1069 was made of. + if !strings.Contains(msg, "pageSize") { + t.Errorf("error %q does not name the widget's own property keys", msg) + } + if deps.saves != 0 { + t.Errorf("validation wrote to storage %d times, want 0", deps.saves) + } +} + +// TestAlterSet_UnknownWidget — the same inversion one level up. Both spellings +// of the mistake used to pass check. +func TestAlterSet_UnknownWidget(t *testing.T) { + ctx, _, _ := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `alter page MyModule.P_Grid { set pageSize = 12 on noSuchWidget; }`) + if len(errs) != 1 { + t.Fatalf("got %d errors, want 1: %v", len(errs), errs) + } + if !strings.Contains(errs[0].Error(), "noSuchWidget") { + t.Errorf("error %q does not name the widget", errs[0]) + } +} + +// TestAlterSet_ReportsEveryBadProperty — a statement setting several properties +// reports all the bad ones, not just the first. exec stops at the first, which +// is right for a run and wrong for a pre-flight: the point of checking is to +// hand back the whole list once. +func TestAlterSet_ReportsEveryBadProperty(t *testing.T) { + ctx, _, _ := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `alter page MyModule.P_Grid { + set NoSuchProperty = 10 on dgProducts; + set AlsoMissing = 11 on dgProducts; + }`) + if len(errs) != 2 { + t.Fatalf("got %d errors, want 2: %v", len(errs), errs) + } +} + +// --------------------------------------------------------------------------- +// Controls — the ways this could block a script that works +// --------------------------------------------------------------------------- + +// TestAlterSet_ValidPropertyInAnyCasing is the control that matters most for +// this area: #1069 was a case-sensitive resolver, and a check that reintroduced +// one would refuse the spelling DESCRIBE PAGE prints. +func TestAlterSet_ValidPropertyInAnyCasing(t *testing.T) { + for _, spelling := range []string{"pageSize", "PageSize", "pagesize", "PAGESIZE"} { + t.Run(spelling, func(t *testing.T) { + ctx, _, _ := gridPageCtx(t) + errs := checkAlterSet(t, ctx, + `alter page MyModule.P_Grid { set `+spelling+` = 12 on dgProducts; }`) + if len(errs) != 0 { + t.Fatalf("valid property %s reported: %v", spelling, errs) + } + }) + } +} + +// TestAlterSet_WidgetTheScriptInserts — a widget an INSERT in the same script +// puts on the page is not in the stored document, and must not be reported as +// missing. +func TestAlterSet_WidgetTheScriptInserts(t *testing.T) { + ctx, _, _ := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `alter page MyModule.P_Grid { + insert into dgProducts { container cNew } + set class = 'x' on cNew; + }`) + if len(errs) != 0 { + t.Fatalf("a widget the script inserts was reported: %v", errs) + } +} + +// TestAlterSet_PageTheScriptCreates — nothing is stored to dry-run against, and +// the widgets the CREATE carries are checked by ValidateWidgetProperties. The +// document must not even be opened. +func TestAlterSet_PageTheScriptCreates(t *testing.T) { + ctx, _, opens := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `create page MyModule.P_New (Title: 'New') { container c1 { } } + alter page MyModule.P_New { set NoSuchProperty = 10 on c1; }`) + if len(errs) != 0 { + t.Fatalf("a page the script creates was reported: %v", errs) + } + if *opens != 0 { + t.Errorf("opened %d documents for a page that is not stored, want 0", *opens) + } +} + +// TestAlterSet_MissingPageIsLeftToTheTargetCheck — validateAlterTarget already +// reports a page that does not exist, in its own wording. Reporting it again +// here reads as two problems. +func TestAlterSet_MissingPageIsLeftToTheTargetCheck(t *testing.T) { + ctx, _, _ := gridPageCtx(t) + + errs := checkAlterSet(t, ctx, `alter page MyModule.P_Missing { set pageSize = 12 on dgProducts; }`) + if len(errs) != 0 { + t.Fatalf("a missing page was reported by this pass: %v", errs) + } +} + +// TestAlterSet_BackendWithoutProbeIsLeftAlone — a backend whose mutator resolves +// properties differently (the MCP one has no pluggable path at all) is left +// unchecked rather than wrongly checked. +func TestAlterSet_BackendWithoutProbeIsLeftAlone(t *testing.T) { + mod := mkModule("MyModule") + pg := mkPage(mod.ID, "P_Grid") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListFoldersFunc: func() ([]*types.FolderInfo, error) { return nil, nil }, + ListPagesFunc: func() ([]*pages.Page, error) { return []*pages.Page{pg}, nil }, + OpenPageForMutationFunc: func(model.ID) (backend.PageMutator, error) { + return &mock.MockPageMutator{}, nil // no Probe method + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + if errs := checkAlterSet(t, ctx, + `alter page MyModule.P_Grid { set NoSuchProperty = 10 on dgProducts; }`); len(errs) != 0 { + t.Fatalf("a backend without the dry-run capability was checked anyway: %v", errs) + } +} + +// TestAlterSet_NotConnected — with no project there is no document, and this +// tier does not run at all. +func TestAlterSet_NotConnected(t *testing.T) { + mb := &mock.MockBackend{IsConnectedFunc: func() bool { return false }} + ctx, _ := newMockCtx(t, withBackend(mb)) + + if errs := checkAlterSet(t, ctx, + `alter page MyModule.P_Grid { set NoSuchProperty = 10 on dgProducts; }`); len(errs) != 0 { + t.Fatalf("ran without a project: %v", errs) + } +} From d7ec1de78c24cb4530499ab47f93d1b0ac1ab1f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 16:11:25 +0000 Subject: [PATCH 09/19] fix(widgets): read a chained ternary's branch condition, not the whole ternary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of Combo box's hide-rules could not be lifted with their full condition. They sit inside `"association"===t.optionsSourceType && ( … )`, itself the ELSE branch of a ternary inside `"context"===t.source ? ( … )` — a chained ternary, where each branch's BODY is parenthesised and each branch's CONDITION is not. groupGuard read the condition of a group opened after `&&`/`||` with trailingExpr, which stops at a STATEMENT separator. A chained ternary contains none, so it handed back the whole `A ? (…) : B` expression as the group's condition. Not a comparison, so guardToCondition refused it and the chain read as unreadable. operandBefore takes lastGuardExpr's answer — it bounds at `:` and `?` as well, giving exactly `"association"===t.optionsSourceType` — but only when it stopped at a boundary INSIDE an expression. lastGuardExpr also bounds at `{`, so where the expression follows a block (ProgressCircle's ternary follows a whole `switch`) it returns a fragment with an unbalanced `}`; there trailingExpr is still right. Hence a choice on the boundary rather than a swap. Measured over all 42 describable widgets: 157 -> 160 of 237 recognized, three rules gained, zero lost, zero per-widget regressions. All three are Combo box's, which goes 21 -> 24 of 32, and each carries its full three-term conjunction — its own guard, the `&&` operand, and the outer branch — so none of them claims hidden where the editor shows the property. Two bindings leave the generated example, both traced to source: with the example's own `source: 'context'`, `optionsSourceType: 'association'`, no association datasource and `selectAllButton: false`, all three terms hold for `optionsSourceAssociation- CaptionType` and for `selectAllButtonCaption`. No binding was added. The regression test takes the real Combo box .mpk. Reverting operandBefore makes it fail naming all three rules. The ceiling this removes was described in PR #427 as "ternary chains without parentheses, which the outward walk does not traverse". That was wrong: instrumenting the walk shows it reaches these groups, and the failure was one function away in guard extraction. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KTdvGwVewkdgNQGDCxpQPZ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + mdl/executor/editorconfig_extract.go | 33 ++++++++++- mdl/executor/editorconfig_shapes_test.go | 56 +++++++++++++++++++ 3 files changed, 89 insertions(+), 1 deletion(-) diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 267a74f52b..9f7e798b52 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -568,3 +568,4 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "`mxcli widget describe` over-listed required bindings because half of each widget's editor visibility rules were never lifted: Combo box reported '16 of 32 editor hide-rules recognized', so properties its editor hides in the described configuration were still asked for. Across the 25 widgets in the fixture that carry rules, 99 of 177 hide-calls (56%) were recognized.", "cause": "Two independent gaps. (1) The guard vocabulary covered only string equality and bare truthiness, so `null===e.dataSource`, minified booleans `!1===e.showFooter`, `0===e.list.length` and `[\"a\",\"b\"].includes(e.type)` all read as unsupported; and a hide that was not FIRST in a `cond ? (hide(a), hide(b))` comma group got no guard at all. (2) The rule model held ONE condition, but editorConfig nests branches — Combo box reaches `source==\"context\" && optionsSourceType==\"association\" && showFooter==false` three levels deep — so a lifted rule could only ever be one conjunct.", "file": "mdl/executor/editorconfig_extract.go, mdl/types/widget_visibility.go", "insight": "Coverage is the wrong thing to optimise on its own: the first cut took Combo box 16->27 and left the describe output BYTE-IDENTICAL, because every consumer skips a rule whose condition value is indeterminable and `exampleValues` records a property only when its default is non-empty — which an unset datasource never is. Moving a metric without moving the outcome is the failure mode to watch for; measure the user-visible artifact (the generated MDL example), not the counter. Worse, several of those 11 new rules were WRONG: they stated one conjunct of a nested condition, and `pagingPosition` (Datagrid) then read as hidden whenever the row count is off, pagination or not — pagination defaults ON, so that is the common case. The fix is to carry the whole conjunction (WidgetVisibilityRule.And) rather than to reject or to guess, and to make every consumer evaluate it (Rule.Fires) — a reader that looks only at HiddenWhen silently over-fires, which is exactly the bug wearing the right field name. Three sub-traps, each found only by diffing rules against the editorConfig source: the innermost enclosing group is often the very group whose guard the hide already carries, so terms must be deduped; the comma walk must not cross a comma that is not a group separator (`A?B:hide(x),hide(y)` gave Maps' `advanced` a guard belonging to a different property); and the PLATFORM argument is not a configuration term — TreeNode gates on `\"web\"===platform`, which MDL always satisfies, so folding it away keeps three bindings correctly pruned. Discipline that made this safe: the acceptance bar was 'zero rules lost, zero regressions, and every removed binding traced to the editorConfig source', not 'the number went up'.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "CI build-and-test fails on a newly added mdl-examples/doctype-tests/ script with `Execution error: ... needs the modelsdk engine (run without MXCLI_ENGINE=legacy)` — while `mxcli check` and a local exec both pass", "cause": "TestMxCheck_DoctypeScripts runs every doctype script through exec + mx check on BOTH engines. A script using a modelsdk-only capability (creating a navigation profile, menu/rule/layout authoring) cannot pass on legacy, where the backend refuses by design rather than approximating the document", "file": "`mdl/executor/roundtrip_doctype_test.go` (engineScriptSkip)", "insight": "A doctype example is a two-engine test, not a one-engine one, and nothing local tells you: `mxcli check` needs no engine and a local exec uses the default (modelsdk). Before adding an example, ask whether anything in it is modelsdk-only — the refusals are deliberate and listed in mdl/backend/mpr/backend.go. The remedy is an engineScriptSkip entry naming WHY the engine refuses, not weakening the script; and note separately whether the feature under test is itself dual-engine, since here only the profile CREATION was modelsdk-only while the SYNC block works on both and is unit-tested on each", "refs": ["ako/mxcli#420"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A widget-describe change measured as 'zero rules lost' had in fact dropped SIX visibility rules from Combo box — the widget the work was justified by. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` (33 of them) but `mxcli widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42; the missing 9 included the one that mattered.", "cause": "The sweep's widget list was derived from an artifact of the pipeline (generated def.json files) rather than from the capability under test (what `widget describe` can describe). Nothing compared the two lists, so the sample silently excluded a whole class — the same 'two lists, nothing comparing them' shape as the defects being fixed.", "file": "mdl/executor/editorconfig_extract.go, mdl/executor/editorconfig_shapes_test.go", "insight": "Derive a sweep's population from the CAPABILITY, never from a convenient artifact, and state the population in the claim: 'zero rules lost across the 33 widgets carrying a def.json' would have been true and would have invited the question. A second trap immediately after: the regression test written to lock the fix used a hand-written editorConfig snippet of the same APPARENT shape, and it passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and the test proved nothing. Pointing the test at the real committed .mpk made the control fail with all six names. Rule of thumb: when a defect was found in real vendor input, the regression test takes the real input; a reconstructed minimal case must be shown to fail without the fix BEFORE it is trusted, and here it did not. The fix itself is the policy that should have been there from the start: conjunction support may WITHHOLD a rule the extractor never produced before (emitting one conjunct over-fires), but may never drop one the older vocabulary already lifted — that rule's accuracy is unchanged by the new work.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "Six of Combo box's editorConfig hide-rules could not be lifted with their full condition, and its coverage counter sat at 21 of 32. The rules sit inside `\"association\"===t.optionsSourceType && ( … )`, itself the ELSE branch of a ternary inside `\"context\"===t.source ? ( … )` — a chained ternary, where each branch's BODY is parenthesised but each branch's CONDITION is not.", "cause": "groupGuard read the condition of a group opened after `&&`/`||` with trailingExpr, which stops at a STATEMENT separator. A chained ternary contains none, so it returned the whole `A ? (…) : B` expression as the group's condition. That is not a comparison, guardToCondition refused it, and enclosingGroupConditions reported the chain unreadable.", "file": "mdl/executor/editorconfig_extract.go (groupGuard, operandBefore)", "insight": "The characterisation written into the PR body — 'ternary chains without parentheses, which the outward walk does not traverse' — was WRONG, and instrumenting the walk rather than re-reading it is what showed so: the walk reaches these groups fine; the failure is guard EXTRACTION at the group, one function away. A one-line ceiling written from reasoning is worth re-deriving before anyone builds on it. The fix is not a straight swap to lastGuardExpr, which bounds at `{` and hands back a fragment with an unbalanced `}` where the expression follows a block (ProgressCircle's ternary follows a whole switch); take lastGuardExpr's answer only when it stopped at an INSIDE-expression boundary (`:`, `?`, `,`) and fall back to trailingExpr otherwise. Also a measurement note: the 'before' number quoted from an earlier session (16 of 32) was stale — the branch had been restarted from a main that already carried the previous fix, so the real baseline was 21. Re-measure the baseline in the tree you are actually editing rather than quoting a figure from memory.", "refs": ["mendixlabs/mxcli#1036"]} diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go index 09fd288b60..f559460d86 100644 --- a/mdl/executor/editorconfig_extract.go +++ b/mdl/executor/editorconfig_extract.go @@ -963,7 +963,7 @@ func groupGuard(js string, callStart int) string { head := strings.TrimRight(js[:open], " ") for _, c := range []string{"&&", "||"} { if strings.HasSuffix(head, c) { - return stripReturnPrefix(trailingExpr(head[:len(head)-2])) + return stripReturnPrefix(operandBefore(head[:len(head)-2])) } } for _, c := range []string{"?", ":"} { @@ -987,6 +987,37 @@ func groupGuard(js string, callStart int) string { return "" } +// operandBefore returns the single expression immediately to the left of a +// connector, which is the group's own condition. +// +// `trailingExpr` alone is too greedy here. It stops at a STATEMENT separator, +// and a chained ternary contains none — so for Combo box's +// +// ["enumeration","boolean"].includes(t.optionsSourceType) +// ? ( …hides… ) +// : "association"===t.optionsSourceType && ( …hides… ) +// +// it returns the whole `A ? (…) : B` expression as the "condition" of the `&&` +// group, which is not a comparison, so the chain reads as unreadable and six +// rules go unlifted. `lastGuardExpr` bounds at `:` and `?` as well and yields +// exactly `"association"===t.optionsSourceType`. +// +// It is not a straight swap: `lastGuardExpr` bounds at `{` too, so where the +// expression is preceded by a block — ProgressCircle's ternary follows a whole +// `switch` — it hands back a fragment with an unbalanced `}`. So take +// lastGuardExpr's answer only when it stopped at a boundary INSIDE an +// expression (`:`, `?`, `,`), and fall back to trailingExpr otherwise. +func operandBefore(head string) string { + guard, boundary := lastGuardExpr(head) + switch boundary { + case ':', '?', ',': + if guard != "" { + return guard + } + } + return trailingExpr(head) +} + // matchingTernaryQuestion returns the index of the `?` matching the `:` that // ends head, skipping over parenthesised groups and nested ternaries. A plain // LastIndexByte finds the innermost `?` instead — in Maps that is a nested diff --git a/mdl/executor/editorconfig_shapes_test.go b/mdl/executor/editorconfig_shapes_test.go index 9b8ac0b56f..d4884ccb62 100644 --- a/mdl/executor/editorconfig_shapes_test.go +++ b/mdl/executor/editorconfig_shapes_test.go @@ -326,3 +326,59 @@ func TestConjunctionWithholdsANewShapeItCannotFullyRead(t *testing.T) { t.Error("no rule survived at all; the withholding is too broad") } } + +// A chained ternary is an else-if ladder: each branch's BODY is parenthesised, +// each branch's CONDITION is not. Combo box nests two of them: +// +// "context"===t.source ? ( …, +// ["enumeration","boolean"].includes(t.optionsSourceType) ? ( … ) +// : "association"===t.optionsSourceType && ( …hides… ) ) +// : "database"===t.source ? ( … ) +// +// The `&&` group's own condition is the operand immediately left of the `&&`. +// Reading back to the nearest STATEMENT separator instead returns the whole +// `A ? (…) : B` expression, which is not a comparison — so the chain read as +// unreadable and these rules went unlifted. +// +// Real .mpk, not a reconstruction: the shape that triggers this is specific +// enough that a hand-written stand-in did not reproduce it last time. +func TestChainedTernaryBranchConditionIsLifted(t *testing.T) { + js, err := mpk.ReadEditorConfig( + "../../testdata/expr-checker/widgets/com.mendix.widget.web.Combobox.mpk", + "com.mendix.widget.web.combobox.Combobox") + if err != nil || js == "" { + t.Fatalf("read Combo box editorConfig: %v", err) + } + rules, _ := extractVisibilityRulesFromJS(js) + + // Each of these sits inside the `"association"===optionsSourceType && (…)` + // group, itself the else branch of a ternary inside the `"context"===source` + // branch. The rule must carry all three terms — its own, the `&&` operand, + // and the outer branch — or it claims hidden where the editor shows it. + want := map[string][]string{ + "menuFooterContent": {"showFooter", "optionsSourceType", "source"}, + "selectAllButtonCaption": {"selectAllButton", "optionsSourceType", "source"}, + "optionsSourceAssociationCaptionType": {"optionsSourceAssociationDataSource", "optionsSourceType", "source"}, + } + for prop, terms := range want { + var found bool + for _, r := range rules { + if r.PropertyKey != prop || r.HiddenWhen == nil || r.HiddenWhen.PropertyKey != terms[0] { + continue + } + found = true + got := map[string]bool{} + for _, c := range r.Conditions() { + got[c.PropertyKey] = true + } + for _, term := range terms { + if !got[term] { + t.Errorf("%s: conjunction is missing %q (has %v)", prop, term, got) + } + } + } + if !found { + t.Errorf("no rule lifted for %s guarded by %s", prop, terms[0]) + } + } +} From 039a1f10af487d4d997631357ea8d1ccb645dd89 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 20:05:41 +0000 Subject: [PATCH 10/19] feat(modelsdk): write SOAP call web service on the codec engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `call web service` was the documented reason the legacy engine still had to exist (cmd/mxcli/engine.go). It was worse than an unsupported feature. The codec engine READ the action already, but microflowActionToGen had no case for it, so it fell through to `default: return nil` and the enclosing ActionActivity was serialized with no action at all. Measured on 11.13.0 before this change: `mxcli exec 06b-soap-examples.mdl` on the DEFAULT engine reported success on all three microflows and `mx check` then reported [CE0008] "No action defined." at Action activity 'Activity' [CE0109] "Undefined variable 'Root'." (x2) the CE0109s being knock-on from the dropped action never binding $Root. That is the #850 shape: a missing WRITE case is a silent drop that exec reports as success. The target is byte-parity with sdk/mpr.serializeWebServiceCallAction, not an independent reading of the metamodel. No Studio Pro-authored SOAP document exists in this repo, so legacy's output is the only reference there is — and it is what users' projects already contain. Parity was established by diffing the two engines' documents from a real project (`mxcli bson dump`, IDs normalised), not by reading the serializer. That diff is what surfaced the three things worth knowing, all about how the codec emits nulls and markers: - A Part property with no child encodes to nil and the encoder then SKIPS the key, so an unset part is an ABSENT key, never a null one. The nulls are carried as primitive bson.Null{} values, which marshal in place. - codec.TypeDefaults.NullFields does emit the key but appends it after every property, so it cannot reproduce alphabetical key order. - NullFields and list markers are registered per $Type, and several types are SHARED between writers: Microflows$HttpConfiguration needs HttpHeaderEntries marker 3 for SOAP and 2 for REST, and legacy writes CustomLocationTemplate as null for SOAP but omits it for REST. A global registration would have silently changed the REST path, so both are written explicitly per call site. Verification: - the two engines' CallWebServiceAction documents are now identical, key for key and value for value - mx check 11.13.0: 3 structural errors -> 0, leaving only the script's 4 deliberate CE1613 dangling refs, which is exactly legacy's output - revert control: removing the switch case fails all six unit tests with the CE0008 message - the modelsdk/06b doctype skip is removed; the script now runs on BOTH engines, and the REST/OData scripts still pass on both Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + cmd/mxcli/engine.go | 7 +- cmd/mxcli/main.go | 2 +- .../modelsdk/microflow_webservice_write.go | 195 ++++++++++++++ .../microflow_webservice_write_test.go | 240 ++++++++++++++++++ mdl/backend/modelsdk/microflow_write.go | 6 + mdl/executor/roundtrip_doctype_test.go | 8 +- 7 files changed, 452 insertions(+), 7 deletions(-) create mode 100644 mdl/backend/modelsdk/microflow_webservice_write.go create mode 100644 mdl/backend/modelsdk/microflow_webservice_write_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index 9f98f60b9f..dec00b9d1e 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -86,3 +86,4 @@ {"area": "mdl/backend", "date": "2026-09-08", "symptom": "An offline navigation profile authored by mxcli builds, routes and installs as a PWA, and shows an empty app — every gate green", "cause": "MDL had no syntax for offline synchronization, so a created offline profile got an empty OfflineEntityConfigs list. A Mendix offline profile downloads nothing until each entity has a sync mode; `mx check` reports 0 errors either way because an empty list is valid", "file": "`mdl/grammar/MDLParser.g4` (navSyncDef), `mdl/backend/modelsdk/navigation_write.go`, `sdk/mpr/writer_navigation.go`", "insight": "Creating a document kind is not the same as being able to configure it, and the gap is invisible to every static check — the symptom is an empty screen at runtime. When adding a profile/document kind, ask what makes it DO anything, not just what makes it exist. The write is an overlay keyed by entity so CompatibilityMode (stored, unauthorable) survives; building the element from the spec alone would clear it silently, the access-rule defect again", "refs": ["ako/TestApp", "PROPOSAL_offline_sync_configuration.md"]} {"area": "mdl/backend", "date": "2026-09-08", "symptom": "`alter page P { set PageSize = 10 on }` errors `pluggable property \"PageSize\" not found` on a grid that `create page \u2026 (PageSize: 20)` had just written and the app really pages at. `mxcli check --references` passes the script, so it fails only at exec, after earlier statements have landed; DESCRIBE PAGE prints the same capitalised `PageSize:`, so describe \u2192 edit \u2192 exec produced a script mxcli refused to run", "cause": "A pluggable property key is lowerCamel in the widget template (`pageSize`). CREATE resolves the author's spelling case-INsensitively (widget engine `lookupProperty`, and `WidgetV3.GetStringProp` before it); ALTER went through `setPluggableWidgetPropertyMut`, which compared the template key byte-for-byte, so only the exact `pageSize` worked. Direct sequel to Findings #1 (2026-07-27), which fixed the same class for FIRST-CLASS props and deliberately left the pluggable fallback case-sensitive with the comment 'template keys must match the template exactly'", "file": "`mdl/backend/pagemutator/mutator.go` (`setPluggableWidgetPropertyMut`)", "insight": "`strings.EqualFold` against the widget's own PropertyTypes. **The disproven belief is the reusable part**: keys are STORED case-sensitively, which is not a reason to MATCH them that way \u2014 the resolver searches one object type's PropertyTypes, and across every shipped template and definition that scope holds no two keys differing only in case (96 scopes, 1208 keys, 0 collisions). Measure the ambiguity before assuming it; here there was none, and the assumption cost a whole verb. **Cheapest localiser**: run the failing statement with the template's own casing \u2014 `set pageSize` succeeded where `set PageSize` failed, in ONE measurement, on the same widget in the same project. Both engines share `pagemutator`, so an engine split says nothing here (verified: modelsdk and legacy both fixed by the one change). Tests `TestSetPluggableProperty_MatchesTemplateKeyRegardlessOfCase` (+ typo-still-errors control, + `TestPluggablePropertyKeysAreUniqueIgnoringCase` pinning the no-collision argument); repro `mdl-examples/bug-tests/alter-page-pluggable-property-casing.mdl`; verified 0 errors on `mx check` 11.13.0. Two reporter claims did NOT hold: `describe page` DOES emit PageSize, but only when it differs from the widget default 20 (deliberate, so describe round-trips) \u2014 at the default it is omitted, which reads as 'describe cannot show it'. Still open and separate: `check --references` does not resolve pluggable property names at all, so a genuine typo (`PagSize`) still checks clean and fails at exec", "refs": ["mendixlabs/mxcli#1069"]} {"area": "mdl/backend", "date": "2026-09-09", "symptom": "A control that should have proven a guard was load-bearing PASSED with the guard removed — the test could not distinguish a correct writer from one that reset the property on every write", "cause": "The fixture stored `false` for a boolean property, and false is also the zero value. A writer that ignored the spec entirely wrote false; the correct writer preserved false. Identical output, so the assertion held either way", "file": "`mdl/backend/modelsdk/navigation_throw_sync_test.go`", "insight": "For a BOOLEAN property, a preservation test must exercise BOTH stored values — the non-zero one is the only case that can fail. This is the second time in one feature: CompatibilityMode needed a synthetic `true` because all seven reference configs carry false. The generalisation: when every real document agrees on a value, the fixture drawn from real documents cannot test preservation, and a synthetic counter-case is not optional. The tell is a control that fails to fail — if stubbing the guard leaves the suite green, the test is measuring nothing, and that is worse than no test because it reads as coverage", "refs": ["ako/mxcli#413", "ThrowPartialSyncError"]} +{"area": "mdl/backend", "date": "2026-09-09", "symptom": "`call web service` (legacy SOAP) on the DEFAULT engine: `mxcli exec 06b-soap-examples.mdl` reported success on all three microflows and `mx check` (11.13.0) then failed the project with `[CE0008] \"No action defined.\" at Action activity 'Activity'` plus two `[CE0109] \"Undefined variable 'Root'.\"`. Only reachable by rerunning with MXCLI_ENGINE=legacy, which is why legacy was still the documented fallback", "cause": "The codec engine READ the action (`actionFromGen` \u2192 `*microflows.WebServiceCallAction`, with a raw fallback) but `microflowActionToGen` had no case for it, so it hit `default: return nil` and the enclosing ActionActivity serialized with no action at all. The #850 shape: a missing WRITE case is not an unsupported feature, it is a silent drop that exec reports as success. The CE0109s are knock-on \u2014 the dropped action never bound $Root", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go` (new), `microflow_write.go` (switch case)", "insight": "**Mirror the legacy serializer, and prove it by diffing the two engines' documents \u2014 do not re-derive the shape from the metamodel.** There is no Studio Pro-authored SOAP document in this repo, so legacy's output is the only reference that exists and is also what users' projects already contain. Method: exec the same script on each engine, `mxcli bson dump` both, normalise the random $IDs, diff. Three discrepancies fell out that no amount of reading would have shown, all in how the codec emits NULLS and MARKERS: (1) a Part property with no child encodes to nil and the encoder then SKIPS the key (`if val != nil`), so an unset part is an ABSENT key, never a null one \u2014 carry the null as a primitive `bson.Null{}` value instead, which marshals in place; (2) `codec.TypeDefaults.NullFields` does emit the key but APPENDS it after every property, so it cannot reproduce alphabetical key order; (3) both NullFields and list markers are registered per `$Type` and several types are SHARED between writers \u2014 `Microflows$HttpConfiguration` needs HttpHeaderEntries marker 3 for SOAP and 2 for REST, and legacy writes CustomLocationTemplate as null for SOAP but omits it for REST, so a global registration would have silently changed the REST path. Write those explicitly per call site. (The package already carries one such collision: `Microflows$HttpHeaderEntry` is registered 2 in microflow_write.go and 3 in odata_write.go, decided by file order.) Verified: the two engines' CallWebServiceAction documents are now identical key-for-key and value-for-value; `mx check` goes 3 structural errors \u2192 0, leaving only the script's 4 deliberate CE1613 dangling refs, which is exactly legacy's output. Revert control: removing the switch case fails all six unit tests with the CE0008 message. The doctype engineScriptSkip for modelsdk/06b was removed and the script now runs on BOTH engines", "refs": []} diff --git a/cmd/mxcli/engine.go b/cmd/mxcli/engine.go index d8c0d17856..2a131646ce 100644 --- a/cmd/mxcli/engine.go +++ b/cmd/mxcli/engine.go @@ -18,7 +18,10 @@ import ( // This is the single selection seam for the engine swap described in // docs/plans/2026-06-05-adopt-modelsdk-engine.md. The codec engine ("modelsdk") // is now the default; "legacy" (sdk/mpr) remains an explicit fallback for the few -// constructs the codec path doesn't yet write (notably SOAP web services). +// constructs the codec path doesn't yet write. SOAP `call web service` is no +// longer among them — it was the last one this comment named, and the codec +// engine now writes it at byte-parity with legacy (mdl/backend/modelsdk/ +// microflow_webservice_write.go). // "compare" is recognised so the contract is stable but still fails fast. type engineKind string @@ -58,7 +61,7 @@ func resolveEngine() engineKind { // modelsdk (default) runs the codec engine: complete reads and writes, validated // at parity with legacy across the doctype suite. The legacy (sdk/mpr) engine is // the explicit fallback for the few constructs the codec path doesn't yet write -// (notably SOAP web services) — select it with --engine legacy or +// — select it with --engine legacy or // MXCLI_ENGINE=legacy. compare needs the run-both diff harness (not yet wired) // and fails fast. An unknown value was already rejected by resolveEngine. func newBackendFactory() func() backend.FullBackend { diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index b93d24414b..5bdd8ef47d 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -317,7 +317,7 @@ func init() { rootCmd.PersistentFlags().Bool("mcp-run", false, "After the command, start the app in Studio Pro via Concord (run_app) and print its URL (requires --mcp-concord)") rootCmd.PersistentFlags().Bool("mcp-verbose", false, "Print each PED tool call the MCP backend makes (requires --mcp)") rootCmd.PersistentFlags().Bool("mcp-trace", false, "Print each MDL command with the PED tool calls it makes (implies --mcp-verbose; requires --mcp)") - rootCmd.PersistentFlags().String("engine", "", "Model engine: modelsdk (default), legacy (fallback for unsupported writes, e.g. SOAP). Overrides MXCLI_ENGINE.") + rootCmd.PersistentFlags().String("engine", "", "Model engine: modelsdk (default), legacy (fallback for unsupported writes). Overrides MXCLI_ENGINE.") rootCmd.Flags().StringP("command", "c", "", "Execute MDL command(s) and exit") // Check command flags diff --git a/mdl/backend/modelsdk/microflow_webservice_write.go b/mdl/backend/modelsdk/microflow_webservice_write.go new file mode 100644 index 0000000000..2b3d80922a --- /dev/null +++ b/mdl/backend/modelsdk/microflow_webservice_write.go @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "strings" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + "github.com/mendixlabs/mxcli/modelsdk/property" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// SOAP `call web service` on the codec engine. +// +// The codec engine READ this action already (microflow_read_actions.go), but the +// write switch had no case for it, so it fell through to `default: return nil` +// and the enclosing ActionActivity was written with NO action at all. That is the +// #850 shape and it is worse than an unsupported feature: `mxcli exec` reported +// success on all three microflows and mxbuild then failed the project — +// measured on 11.13.0, `06b-soap-examples.mdl` on the DEFAULT engine gives +// +// [CE0008] "No action defined." at Action activity 'Activity' +// [CE0109] "Undefined variable 'Root'." at End event (×2) +// +// the CE0109s being the knock-on from the dropped action never binding $Root. +// Legacy was the documented fallback (cmd/mxcli/engine.go), which is exactly the +// dependency that keeps the legacy engine alive. +// +// The target shape is byte-parity with the legacy serializer +// (sdk/mpr.serializeWebServiceCallAction), not an independent reading of the +// metamodel. There is no Studio Pro-authored SOAP document in this repo to pin +// against, so legacy's output is the only reference that exists — and it is what +// users' projects already contain. TestWebServiceCallAction_MatchesLegacyBSON +// holds the two engines together; a discrepancy is a test failure, not a silent +// divergence. +// +// Two shapes are deliberately NOT re-derived here: +// +// - HttpHeaderEntries is written as an empty typed array with marker 3, and +// the two SimpleRequestHandling ParameterMappings with marker 2, EXPLICITLY +// rather than through codec.RegisterTypeDefaults. Those registrations are +// global and keyed by $Type, and Microflows$HttpConfiguration is shared with +// the REST writer, which needs marker 2 for the same field — registering a +// SOAP-shaped default would silently change REST's output. (The package +// already carries one such collision: Microflows$HttpHeaderEntry is +// registered as 2 in microflow_write.go and as 3 in odata_write.go, and +// which one wins is decided by file order.) +// - RequestBodyHandling is always SimpleRequestHandling, even when the +// statement carries a SEND MAPPING. Legacy does the same and says why: the +// advanced form needs a Studio Pro-generated example to establish its type +// storage name. Writing a guessed $Type is the failure mode that makes a +// project impossible to OPEN rather than merely invalid, so the send mapping +// stays unwritten here exactly as it does on legacy. `call web service raw` +// is the escape hatch for operations that need it. +// +// Every null the document carries is written IN KEY POSITION rather than through +// NullFields, for the same reason and with the same consequence — see addNull. + +// webServiceCallActionToGen builds a Microflows$CallWebServiceAction. Mirrors +// sdk/mpr.serializeWebServiceCallAction field-for-field, in the same key order. +func webServiceCallActionToGen(a *microflows.WebServiceCallAction) element.Element { + // The raw escape hatch: `call web service raw ''` carries an opaque + // document that must re-emit byte-for-byte. Legacy returns the unmarshalled + // payload verbatim; the codec's decoder preserves an unknown subtree the same + // way, so this is passthrough on both engines. + if len(a.RawBSON) > 0 { + if el, err := codec.NewDecoder(codec.DefaultRegistry).Decode(a.RawBSON); err == nil && el != nil { + return el + } + // A payload that will not decode is not silently dropped — falling + // through writes the structured form, which is wrong but visible, and + // the executor already reports a bad base64 payload at build time. + } + + g := newElem("Microflows$CallWebServiceAction", string(a.ID)) + addStr(g, "ErrorHandlingType", orDefault(string(a.ErrorHandlingType), "Rollback")) + addPart(g, "HttpConfiguration", webServiceHttpConfigToGen()) + // ImportedService is a BY_NAME_REFERENCE qualified-name string, not a binary + // UUID — the same convention the legacy writer notes. + addStr(g, "ImportedService", string(a.ServiceID)) + addBool(g, "IsValidationRequired", false) + addPart(g, "NewResultHandling", webServiceResultHandlingToGen(a)) + addStr(g, "OperationName", a.OperationName) + addNull(g, "ProxyConfiguration") + addPart(g, "RequestBodyHandling", simpleRequestHandlingToGen()) + addPart(g, "RequestHeaderHandling", simpleRequestHandlingToGen()) + addStr(g, "RequestProxyType", "DefaultProxy") + addStr(g, "ServiceName", webServiceLocalName(string(a.ServiceID))) + addStr(g, "TimeOutExpression", orDefault(a.TimeoutExpression, "300")) + addBool(g, "UseRequestTimeOut", true) + return g +} + +// webServiceHttpConfigToGen builds the fixed HttpConfiguration a SOAP call +// carries. It is not httpConfigToGen: that one is driven by a REST action's own +// configuration and writes OverrideLocation TRUE, while a SOAP call writes the +// defaults with OverrideLocation false. Same $Type, different content. +func webServiceHttpConfigToGen() element.Element { + hc := newElem("Microflows$HttpConfiguration", "") + addStr(hc, "ClientCertificate", "") + addStr(hc, "CustomLocation", "") + addNull(hc, "CustomLocationTemplate") + addStr(hc, "HttpAuthenticationPassword", "") + addStr(hc, "HttpAuthenticationUserName", "") + addEmptyTypedList(hc, "HttpHeaderEntries", 3) + addStr(hc, "HttpMethod", "Post") + addBool(hc, "OverrideLocation", false) + addBool(hc, "UseHttpAuthentication", false) + return hc +} + +// webServiceResultHandlingToGen builds NewResultHandling, a Microflows$ResultHandling +// (the same type REST result handling uses). Bind is driven by whether the +// statement assigned an output variable, and the ImportMappingCall carries the +// RECEIVE mapping — by qualified name, not by UUID. +func webServiceResultHandlingToGen(a *microflows.WebServiceCallAction) element.Element { + rh := newElem("Microflows$ResultHandling", "") + addBool(rh, "Bind", a.OutputVariable != "") + + if a.ReceiveMappingID != "" { + imc := newElem("Microflows$ImportMappingCall", "") + addStr(imc, "Commit", "YesWithoutEvents") + addStr(imc, "ContentType", "Json") + addBool(imc, "ForceSingleOccurrence", false) + addStr(imc, "ObjectHandlingBackup", "Create") + addStr(imc, "ParameterVariableName", "") + rng := newElem("Microflows$ConstantRange", "") + addBool(rng, "SingleObject", true) + addPart(imc, "Range", rng) + // STORAGE NAME: ReturnValueMapping, not "Mapping" — the same key the + // import-mapping call uses everywhere else in this engine. + addStr(imc, "ReturnValueMapping", string(a.ReceiveMappingID)) + addPart(rh, "ImportMappingCall", imc) + } else { + addNull(rh, "ImportMappingCall") + } + + addStr(rh, "ResultVariableName", a.OutputVariable) + addPart(rh, "VariableType", newElem("DataTypes$VoidType", "")) + return rh +} + +// simpleRequestHandlingToGen builds the Microflows$SimpleRequestHandling used for +// both the body and the header handling. +func simpleRequestHandlingToGen() element.Element { + rh := newElem("Microflows$SimpleRequestHandling", "") + addStr(rh, "NullValueOption", "LeaveOutElement") + addEmptyTypedList(rh, "ParameterMappings", 2) + return rh +} + +// webServiceLocalName is the service's local name — the part after the last dot +// of the qualified name. Mendix stores both: ImportedService is qualified, +// ServiceName is not. +func webServiceLocalName(qualified string) string { + if i := strings.LastIndex(qualified, "."); i >= 0 { + return qualified[i+1:] + } + return qualified +} + +// addEmptyTypedList writes an empty typed array carrying an explicit version +// marker. +// +// The marker is Mendix's array version and it is load-bearing — a wrong one is +// the class of defect that makes a project Studio Pro cannot open. It is written +// here rather than registered because the registry is keyed by $Type and these +// parent types are shared with writers that need different markers for the same +// field; see the note at the top of this file. +func addEmptyTypedList(b *element.Base, name string, marker int32) { + p := property.NewPrimitive[bson.A](name, func(bson.Raw, string) bson.A { return nil }) + b.AddProperty(p, uint(len(b.Properties()))) + p.Set(bson.A{marker}) +} + +// addNull writes an explicit BSON null IN KEY POSITION for a child slot the +// document must carry but this call leaves unset. +// +// It cannot be a Part property: the encoder returns nil for a part with no +// child and the caller then skips the key entirely (encoder.go, `if val != nil`), +// so an unset part is an ABSENT key, not a null one. codec.TypeDefaults' +// NullFields does emit the key, but appends it after every property, and it is +// registered per $Type — Microflows$HttpConfiguration is shared with the REST +// writer, whose legacy counterpart writes CustomLocationTemplate only when a +// template exists (sdk/mpr writer_microflow_actions.go:688 vs :794). Registering +// it would add a null REST does not write. So the null is carried as a primitive +// value, which the driver marshals in place. +func addNull(b *element.Base, name string) { + p := property.NewPrimitive[bson.Null](name, func(bson.Raw, string) bson.Null { return bson.Null{} }) + b.AddProperty(p, uint(len(b.Properties()))) + p.Set(bson.Null{}) +} diff --git a/mdl/backend/modelsdk/microflow_webservice_write_test.go b/mdl/backend/modelsdk/microflow_webservice_write_test.go new file mode 100644 index 0000000000..639853dd69 --- /dev/null +++ b/mdl/backend/modelsdk/microflow_webservice_write_test.go @@ -0,0 +1,240 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + bsonv1 "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// encodeMicroflowAction encodes a semantic microflow action through the codec +// engine's write path, the same way a real CREATE MICROFLOW does. +func encodeMicroflowAction(t *testing.T, a microflows.MicroflowAction) bsonv1.D { + t.Helper() + el := microflowActionToGen(a) + if el == nil { + t.Fatal("actionToGen returned nil — the activity would be written with NO action (CE0008)") + } + out, err := (&codec.Encoder{}).Encode(el) + if err != nil { + t.Fatalf("encode: %v", err) + } + var doc bsonv1.D + if err := bsonv1.Unmarshal([]byte(out), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + return doc +} + +func fullWebServiceCall() *microflows.WebServiceCallAction { + return µflows.WebServiceCallAction{ + BaseElement: model.BaseElement{ID: "ws-1"}, + ErrorHandlingType: "Rollback", + ServiceID: "SampleSOAP.OrderService", + OperationName: "FetchSampleItems", + ReceiveMappingID: "SampleSOAP.OrderResponse", + OutputVariable: "Root", + UseReturnVariable: true, + TimeoutExpression: "30", + } +} + +// TestWebServiceCallAction_IsWritten is the regression test for the reported +// gap, and it is the one that fails without the fix. +// +// The codec engine READ this action but had no write case, so it fell through to +// `default: return nil` and the enclosing ActionActivity was serialized with no +// action at all. Measured on 11.13.0 before the fix: `mxcli exec` of +// 06b-soap-examples.mdl reported success on all three microflows and `mx check` +// then reported CE0008 "No action defined." plus two CE0109 "Undefined variable +// 'Root'." — the knock-on from the dropped action never binding the variable. +func TestWebServiceCallAction_IsWritten(t *testing.T) { + if el := microflowActionToGen(fullWebServiceCall()); el == nil { + t.Fatal("microflowActionToGen(*WebServiceCallAction) = nil; the activity would carry no action (CE0008)") + } +} + +// TestWebServiceCallAction_MatchesLegacyDocument pins the whole document against +// the shape the legacy serializer writes. +// +// Legacy is the reference on purpose: there is no Studio Pro-authored SOAP +// document in this repo, and legacy's output is both the documented fallback and +// what users' projects already contain. The values below were read off a real +// legacy-written project (`mxcli bson dump`, Mendix 11.13.0), not off the +// serializer's source. +func TestWebServiceCallAction_MatchesLegacyDocument(t *testing.T) { + doc := encodeMicroflowAction(t, fullWebServiceCall()) + + for _, want := range []struct { + key string + val any + }{ + {"$Type", "Microflows$CallWebServiceAction"}, + {"ErrorHandlingType", "Rollback"}, + // Qualified for ImportedService, local-only for ServiceName. Mendix + // stores both, and they are not the same string. + {"ImportedService", "SampleSOAP.OrderService"}, + {"ServiceName", "OrderService"}, + {"IsValidationRequired", false}, + {"OperationName", "FetchSampleItems"}, + {"RequestProxyType", "DefaultProxy"}, + {"TimeOutExpression", "30"}, + {"UseRequestTimeOut", true}, + {"ProxyConfiguration", nil}, + } { + if got := docGet(doc, want.key); got != want.val { + t.Errorf("%s = %#v, want %#v", want.key, got, want.val) + } + } + + // HttpConfiguration: a SOAP call writes the defaults with OverrideLocation + // FALSE, unlike the REST writer's shared $Type which writes true. + hc, ok := docGet(doc, "HttpConfiguration").(bsonv1.D) + if !ok { + t.Fatalf("HttpConfiguration = %#v, want a document", docGet(doc, "HttpConfiguration")) + } + if got := docGet(hc, "HttpMethod"); got != "Post" { + t.Errorf("HttpConfiguration.HttpMethod = %#v, want Post", got) + } + if got := docGet(hc, "OverrideLocation"); got != false { + t.Errorf("HttpConfiguration.OverrideLocation = %#v, want false", got) + } + // The typed-array marker is load-bearing: a wrong one is the class of defect + // that makes a project Studio Pro cannot open. SOAP's empty HttpHeaderEntries + // is marker 3 where REST's is 2, on the same $Type — which is why this is + // written explicitly rather than registered globally. + assertTypedArrayMarker(t, hc, "HttpHeaderEntries", 3) + + for _, field := range []string{"RequestBodyHandling", "RequestHeaderHandling"} { + rh, ok := docGet(doc, field).(bsonv1.D) + if !ok { + t.Fatalf("%s = %#v, want a document", field, docGet(doc, field)) + } + if got := docGet(rh, "$Type"); got != "Microflows$SimpleRequestHandling" { + t.Errorf("%s.$Type = %#v", field, got) + } + if got := docGet(rh, "NullValueOption"); got != "LeaveOutElement" { + t.Errorf("%s.NullValueOption = %#v", field, got) + } + assertTypedArrayMarker(t, rh, "ParameterMappings", 2) + } +} + +// TestWebServiceCallAction_ResultHandlingBindsTheReceiveMapping — the receive +// mapping travels in the ImportMappingCall under ReturnValueMapping (NOT +// "Mapping", which is what gen binds), by qualified name rather than by UUID. +// Getting the key wrong here reads back as a call with no mapping. +func TestWebServiceCallAction_ResultHandlingBindsTheReceiveMapping(t *testing.T) { + doc := encodeMicroflowAction(t, fullWebServiceCall()) + + rh, ok := docGet(doc, "NewResultHandling").(bsonv1.D) + if !ok { + t.Fatalf("NewResultHandling = %#v, want a document", docGet(doc, "NewResultHandling")) + } + if got := docGet(rh, "Bind"); got != true { + t.Errorf("Bind = %#v, want true (the statement assigned $Root)", got) + } + if got := docGet(rh, "ResultVariableName"); got != "Root" { + t.Errorf("ResultVariableName = %#v, want Root", got) + } + imc, ok := docGet(rh, "ImportMappingCall").(bsonv1.D) + if !ok { + t.Fatalf("ImportMappingCall = %#v, want a document", docGet(rh, "ImportMappingCall")) + } + if got := docGet(imc, "ReturnValueMapping"); got != "SampleSOAP.OrderResponse" { + t.Errorf("ReturnValueMapping = %#v, want the qualified mapping name", got) + } + rng, ok := docGet(imc, "Range").(bsonv1.D) + if !ok { + t.Fatalf("Range = %#v, want a document", docGet(imc, "Range")) + } + if got := docGet(rng, "$Type"); got != "Microflows$ConstantRange" { + t.Errorf("Range.$Type = %#v", got) + } +} + +// TestWebServiceCallAction_NoOutputVariable — a call that binds nothing writes +// Bind false and an explicitly null ImportMappingCall, rather than omitting the +// result handling. +func TestWebServiceCallAction_NoOutputVariable(t *testing.T) { + a := fullWebServiceCall() + a.OutputVariable = "" + a.UseReturnVariable = false + a.ReceiveMappingID = "" + + rh, ok := docGet(encodeMicroflowAction(t, a), "NewResultHandling").(bsonv1.D) + if !ok { + t.Fatal("NewResultHandling missing") + } + if got := docGet(rh, "Bind"); got != false { + t.Errorf("Bind = %#v, want false", got) + } + if got := docGet(rh, "ImportMappingCall"); got != nil { + t.Errorf("ImportMappingCall = %#v, want null", got) + } +} + +// TestWebServiceCallAction_DefaultTimeout — an omitted TIMEOUT writes Mendix's +// own default rather than an empty expression, matching legacy. +func TestWebServiceCallAction_DefaultTimeout(t *testing.T) { + a := fullWebServiceCall() + a.TimeoutExpression = "" + + if got := docGet(encodeMicroflowAction(t, a), "TimeOutExpression"); got != "300" { + t.Errorf("TimeOutExpression = %#v, want 300", got) + } +} + +// TestWebServiceCallAction_RawPassthrough — `call web service raw ''` is +// the escape hatch for operations neither engine can spell structurally (a SEND +// MAPPING needs Mendix$AdvancedRequestHandling, whose storage name has no +// reference here). The payload must re-emit unchanged, not be replaced by the +// structured form. +func TestWebServiceCallAction_RawPassthrough(t *testing.T) { + raw, err := bsonv1.Marshal(bsonv1.D{ + {Key: "$Type", Value: "Microflows$CallWebServiceAction"}, + {Key: "ImportedService", Value: "Raw.Service"}, + {Key: "OperationName", Value: "RawOp"}, + }) + if err != nil { + t.Fatalf("marshal: %v", err) + } + a := fullWebServiceCall() + a.RawBSON = raw + + doc := encodeMicroflowAction(t, a) + if got := docGet(doc, "ImportedService"); got != "Raw.Service" { + t.Errorf("ImportedService = %#v, want the RAW payload's value, not the structured one", got) + } + if got := docGet(doc, "OperationName"); got != "RawOp" { + t.Errorf("OperationName = %#v, want the RAW payload's value", got) + } + // Control: without the raw payload the same action writes the structured + // form, so the assertions above are not satisfied by a writer that ignores + // both. + a.RawBSON = nil + if got := docGet(encodeMicroflowAction(t, a), "ImportedService"); got != "SampleSOAP.OrderService" { + t.Errorf("structured ImportedService = %#v, want SampleSOAP.OrderService", got) + } +} + +// assertTypedArrayMarker checks an empty Mendix typed array: a one-element BSON +// array holding just the int32 version marker. +func assertTypedArrayMarker(t *testing.T, doc bsonv1.D, key string, want int32) { + t.Helper() + arr, ok := docGet(doc, key).(bsonv1.A) + if !ok { + t.Fatalf("%s = %#v, want a typed array", key, docGet(doc, key)) + } + if len(arr) != 1 { + t.Fatalf("%s has %d entries, want just the marker", key, len(arr)) + } + if got, ok := arr[0].(int32); !ok || got != want { + t.Errorf("%s marker = %#v, want int32(%d)", key, arr[0], want) + } +} diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index 3f175cfc64..de10eb78c5 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -852,6 +852,12 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // "call external action" — Microflows$CallExternalAction. Without this // the activity serialized with no action → CE0008 "No action defined". return callExternalActionToGen(a) + case *microflows.WebServiceCallAction: + // "call web service" (legacy SOAP) — Microflows$CallWebServiceAction. + // Same CE0008 shape as the two cases above, and the reason the legacy + // engine was still the documented fallback for SOAP. See + // microflow_webservice_write.go. + return webServiceCallActionToGen(a) default: return nil // not yet supported (added in later groups) } diff --git a/mdl/executor/roundtrip_doctype_test.go b/mdl/executor/roundtrip_doctype_test.go index 6ac41b2146..4a61bd1849 100644 --- a/mdl/executor/roundtrip_doctype_test.go +++ b/mdl/executor/roundtrip_doctype_test.go @@ -43,10 +43,10 @@ var gateEngines = []gateEngine{ // (e.g. a known modelsdk gap on a specific document type). A script broken on // BOTH engines belongs in scriptSkipList instead. Key format: "/". var engineScriptSkip = map[string]string{ - // SOAP web-service calls aren't serialized by the codec engine yet — legacy - // is the documented fallback for SOAP (cmd/mxcli/engine.go). On modelsdk the - // `call web service` activity serializes with no action → CE0008/CE0109. - "modelsdk/06b-soap-examples.mdl": "modelsdk doesn't write SOAP web-service calls yet (legacy fallback); tracked", + // (modelsdk/06b-soap-examples.mdl was skipped here until the codec engine + // learned to write Microflows$CallWebServiceAction. It ran on BOTH engines + // from then on, which is the point of the removal: SOAP was the documented + // reason the legacy engine still had to exist.) // The legacy widget builder has no `barchart` pluggable-widget template, so // page build fails ("template not found: barchart"). Passes on modelsdk. "legacy/34-chart-widget-examples.mdl": "legacy widget builder lacks the barchart template (works on modelsdk); tracked", From 66c24aae04ab8a941aa5785eb7fb61f208ed7d98 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 20:05:49 +0000 Subject: [PATCH 11/19] fix(microflow): stop DESCRIBE dropping an activity's error handler (mendixlabs/mxcli#1078) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A microflow whose activity carried a custom error handler came back from `describe microflow` with the handler — and every activity in its branch — missing. The output stayed valid MDL and `mxcli check` passed, so a describe→edit→exec cycle deleted the handler from the model with nothing to warn you. Reported on a create-variable activity set to "custom with rollback"; reproduced on HEAD against mxbuild 11.14.0 with a pure mxcli round trip, no Studio Pro involved. Two stacked defects, which is why fixing one looked like fixing both. 1. The describer, on both engines. getActionErrorHandlingType was a hand-maintained switch that had drifted to 17 of the 38 action types that store ErrorHandlingType. emitActivityStatement walks an activity's error branch only when hasCustomErrorHandler() agrees, so each of the 21 missing types lost the ENTIRE `on error { … }` block rather than just the suffix — CreateObject and ChangeObject among them. Replaced with a reflection lookup on the field, so a new action type is covered the moment it exists. RestOperationCallAction stays excluded on purpose: Mendix rejects a custom handler there (CE6035). 2. The legacy parser. Nine parse functions never read ErrorHandlingType off the BSON, so the value was gone before the describer could be asked. After fix 1 the default engine round-tripped and legacy still dropped the handler; both engines now agree byte for byte. Fixing the describer alone would have made things worse. Eight statement forms had no onErrorClause in the grammar — declare (the reporter's own), set, change, log, show page, close page, show message, validation feedback — so DESCRIBE began emitting `declare $name String = 'v' on error { … };`, which fails to parse. Trading a silent drop for a broken script is not a fix, so those eight now accept the clause and the round trip is real. Measured on 11.14.0, one microflow per row, and the result is not guessable: all eight accept a custom handler at 0 errors, but for `on error continue` create-variable and change-variable are fine while change-OBJECT, log, show page, close page, show message and validation feedback are CE6035. Those are now in MDL076's deny-list, whose comment claiming Log and Change were "unreachable from a script" this change invalidated. New MDL077 refuses `on error` on the list-operation and aggregate forms of `set`, which genuinely have no ErrorHandlingType in the metamodel — one MDL keyword spanning activities that can and cannot hold the clause. A non-terminating handler that merges back into the main flow and leaves a later variable out of scope on the error path (CE0108) is not a bug here; Studio Pro reports the same for the same model. Controls: revert the lookup and all seven describer cases fail naming the dropped branch; revert one parser and the legacy case fails; a microflow with no clause must still render no suffix (#840 in reverse). Verified end-to-end — describe→exec→describe is byte-identical and reports "Unchanged microflow", and the repro script adds 0 errors over baseline on mxbuild 11.14.0. go build, go test ./... 84 ok / 0 fail, make lint clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../skills/mendix/write-microflows/SKILL.md | 30 +++ cmd/mxcli/syntax/features_microflow.go | 25 ++- docs/01-project/MDL_QUICK_REFERENCE.md | 2 +- ...microflow-1078-error-handler-roundtrip.mdl | 156 ++++++++++++++ mdl/ast/ast_microflow.go | 61 ++++-- .../cmd_microflows_builder_actions.go | 24 ++- mdl/executor/cmd_microflows_builder_calls.go | 40 +++- mdl/executor/cmd_microflows_show_helpers.go | 89 ++++---- .../microflow_error_handler_authoring_test.go | 197 +++++++++++++++++ .../microflow_error_handler_roundtrip_test.go | 198 ++++++++++++++++++ mdl/executor/validate_microflow.go | 24 +++ .../validate_microflow_error_handling.go | 80 ++++++- mdl/grammar/domains/MDLMicroflow.g4 | 16 +- mdl/visitor/visitor_microflow_actions.go | 20 ++ mdl/visitor/visitor_microflow_statements.go | 45 +++- sdk/mpr/parser_microflow.go | 12 ++ sdk/mpr/parser_microflow_actions.go | 15 ++ ...rser_microflow_error_handling_1078_test.go | 96 +++++++++ 19 files changed, 1039 insertions(+), 92 deletions(-) create mode 100644 mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl create mode 100644 mdl/executor/microflow_error_handler_authoring_test.go create mode 100644 mdl/executor/microflow_error_handler_roundtrip_test.go create mode 100644 sdk/mpr/parser_microflow_error_handling_1078_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 603291ee00..9552e24336 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -572,3 +572,4 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A widget-describe change measured as 'zero rules lost' had in fact dropped SIX visibility rules from Combo box — the widget the work was justified by. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` (33 of them) but `mxcli widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42; the missing 9 included the one that mattered.", "cause": "The sweep's widget list was derived from an artifact of the pipeline (generated def.json files) rather than from the capability under test (what `widget describe` can describe). Nothing compared the two lists, so the sample silently excluded a whole class — the same 'two lists, nothing comparing them' shape as the defects being fixed.", "file": "mdl/executor/editorconfig_extract.go, mdl/executor/editorconfig_shapes_test.go", "insight": "Derive a sweep's population from the CAPABILITY, never from a convenient artifact, and state the population in the claim: 'zero rules lost across the 33 widgets carrying a def.json' would have been true and would have invited the question. A second trap immediately after: the regression test written to lock the fix used a hand-written editorConfig snippet of the same APPARENT shape, and it passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and the test proved nothing. Pointing the test at the real committed .mpk made the control fail with all six names. Rule of thumb: when a defect was found in real vendor input, the regression test takes the real input; a reconstructed minimal case must be shown to fail without the fix BEFORE it is trusted, and here it did not. The fix itself is the policy that should have been there from the start: conjunction support may WITHHOLD a rule the extractor never produced before (emitting one conjunct over-fires), but may never drop one the older vocabulary already lifted — that rule's accuracy is unchanged by the new work.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "Six of Combo box's editorConfig hide-rules could not be lifted with their full condition, and its coverage counter sat at 21 of 32. The rules sit inside `\"association\"===t.optionsSourceType && ( … )`, itself the ELSE branch of a ternary inside `\"context\"===t.source ? ( … )` — a chained ternary, where each branch's BODY is parenthesised but each branch's CONDITION is not.", "cause": "groupGuard read the condition of a group opened after `&&`/`||` with trailingExpr, which stops at a STATEMENT separator. A chained ternary contains none, so it returned the whole `A ? (…) : B` expression as the group's condition. That is not a comparison, guardToCondition refused it, and enclosingGroupConditions reported the chain unreadable.", "file": "mdl/executor/editorconfig_extract.go (groupGuard, operandBefore)", "insight": "The characterisation written into the PR body — 'ternary chains without parentheses, which the outward walk does not traverse' — was WRONG, and instrumenting the walk rather than re-reading it is what showed so: the walk reaches these groups fine; the failure is guard EXTRACTION at the group, one function away. A one-line ceiling written from reasoning is worth re-deriving before anyone builds on it. The fix is not a straight swap to lastGuardExpr, which bounds at `{` and hands back a fragment with an unbalanced `}` where the expression follows a block (ProgressCircle's ternary follows a whole switch); take lastGuardExpr's answer only when it stopped at an INSIDE-expression boundary (`:`, `?`, `,`) and fall back to trailingExpr otherwise. Also a measurement note: the 'before' number quoted from an earlier session (16 of 32) was stale — the branch had been restarted from a main that already carried the previous fix, so the real baseline was 21. Re-measure the baseline in the tree you are actually editing rather than quoting a figure from memory.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "`alter page P { set NoSuchProperty = 10 on dgProducts; }` and `set PageSize = 12 on noSuchWidget;` both passed `mxcli check -p app.mpr --references` (exit 0, \"All references valid\") and were then refused by `exec`: `pluggable property \"NoSuchProperty\" not found` / `widget \"noSuchWidget\" not found`. exec applies statements one at a time, so the script had already written everything before the typo", "cause": "ValidateWidgetProperties resolves the properties of widgets a statement CARRIES \u2014 CREATE PAGE's tree, and ALTER's InsertWidgetOp/ReplaceWidgetOp trees. SetPropertyOp carries no widget: it names one already stored, so its property can only be resolved against the DOCUMENT, which that pass never opens. Same inversion validate_alter_target.go closed for the ALTER's target document, one level further in", "file": "`mdl/executor/validate_alter_set.go` (new), `mdl/backend/pagemutator/probe.go` (new)", "insight": "**Do not re-derive what a mutation accepts \u2014 run the mutation.** The vocabulary of an ALTER SET is partly a switch in `setRawWidgetPropertyMut` and partly the STORED widget's own PropertyTypes, which belong to whatever widget package the project installed; no registry in this repo can state it for an arbitrary project. So the check opens the document, runs the real setter against a throwaway deep copy (`Mutator.Probe`, whose `Save` is refused), and keeps only the error \u2014 check and exec cannot drift because there is one resolver. The author also gets exec's exact wording from the pre-flight. **Two false-positive sources, both measured, both silence rather than a finding**: a page the script CREATEs (nothing stored \u2014 skip, and do not even open it) and a widget an INSERT in the same script adds. The second cannot be a name match: a DataGrid 2 column is inserted as `colBrand` and addressed as `Brand` (derived from the bound attribute), so the rule is `ResolvesTarget` \u2014 suppress only when the document does not carry the target AND the script adds widgets to that document. **Gating on an optional interface assertion, not on backend.PageMutator**, keeps the pass off the MCP mutator, which has no pluggable path and would have reported its own difference as the author's mistake. Controls: 4 casings of a real property pass (a case-sensitive pre-flight would have re-broken #1069); insert-then-set passes check AND exec in both the same-statement and across-statement forms; 17 ALTER SET examples in mdl-examples show 0 new errors vs the baseline binary; the project's files are byte-identical (md5 over .mpr + mprcontents) after five check runs. Revert control: stubbing the pass makes the three gap tests fail", "refs": []} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "A microflow whose activity has a CUSTOM error handler comes back from `describe microflow` with the handler AND every activity in its branch missing. Output stays valid MDL, `mxcli check` passes, nothing on stderr \u2014 so describe\u2192edit\u2192exec deletes the handler from the model. Reported on a create-variable activity with \"custom with rollback\" (v0.21, Mx 11.12.3); reproduced on HEAD/11.14.0 with a pure mxcli round trip", "cause": "TWO STACKED DEFECTS. (1) `getActionErrorHandlingType` was a hand-maintained switch covering 17 of the 38 action types that store ErrorHandlingType. `emitActivityStatement` walks the error branch only when `hasCustomErrorHandler(errType)` agrees, so each of the 21 missing types lost the ENTIRE `on error { \u2026 }` block, not just the suffix \u2014 CreateObject and ChangeObject among them. (2) legacy only: 9 parse functions never read ErrorHandlingType off the BSON, so the value was gone before the describer was asked. Fixing (1) alone left legacy still broken, which is how they hid each other", "file": "`mdl/executor/cmd_microflows_show_helpers.go` (`getActionErrorHandlingType` \u2192 new `actionErrorHandlingField`); `sdk/mpr/parser_microflow.go`, `sdk/mpr/parser_microflow_actions.go` (9 parsers); grammar+visitor+builder for 8 statements; `mdl/executor/validate_microflow_error_handling.go` (MDL076 table, new MDL077)", "insight": "**Count the gap before fixing the instance** \u2014 the reported activity was 1 of 21, measured by diffing the switch's cases against the action types declaring the field. **Replace the enumeration, do not extend it**: the list had already been patched per-instance (#863) and silently regrew, so it is now a reflection lookup on the `ErrorHandlingType` field, with `RestOperationCallAction` the ONE deliberate exclusion (Mendix rejects a custom handler there, CE6035). Actions embed model.BaseElement, which has no such field, so no promoted field is picked up by accident. **The trap: fixing the describer alone makes things WORSE.** 8 statement forms had no `onErrorClause` in the grammar \u2014 `declare` (the reporter's own), `set`, `change`, `log`, `show page`, `close page`, `show message`, `validation feedback` \u2014 so DESCRIBE began emitting `declare $name String = 'v' on error { \u2026 };`, which fails to parse (`mismatched input 'on' expecting ';'`). Trading a silent drop for a broken script is not a fix; the grammar was extended instead. **Measured on 11.14.0, and the result is not guessable**: all 8 accept a custom handler (0 errors), but for `on error continue` create-VARIABLE and change-VARIABLE are fine while change-OBJECT, log, show page, close page, show message and validation feedback are CE6035 \u2014 now in MDL076's deny-list, whose comment claiming Log/Change were \"unreachable from a script\" this change invalidated. New MDL077 refuses `on error` on the list-operation/aggregate forms of `set`, which genuinely have no ErrorHandlingType in the metamodel \u2014 one MDL keyword spanning activities that can and cannot hold the clause. **A non-terminating handler causing CE0108 is NOT a bug**: the branch merges back and a later variable is out of scope on the error path; Studio Pro reports the same. Controls: revert the lookup \u2192 all 7 describer cases fail naming the dropped branch; revert one parser \u2192 the legacy case fails; a no-clause microflow must still render NO suffix (#840 in reverse). Repro `mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl`; describe\u2192exec\u2192describe byte-identical and reported \"Unchanged microflow\"", "refs": ["#1078", "#863", "#840"], "ce": ["CE6035", "CE0108"]} diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 6fb425a434..54925e55f4 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -634,3 +634,33 @@ call microflow ... on error rollback; -- Rollback on error call microflow ... on error { log ...; return ...; }; -- Custom handler call microflow ... on error without rollback { ... }; -- No rollback ``` + +The clause goes on whichever activity may fail, not only on calls: + +```mdl +declare $Name String = 'default' on error { return 'could not initialise'; }; +$Name = $Other/Name on error { return 'lookup failed'; }; +change $Order (Status = Shipped) on error { log error 'could not ship'; return; }; +log info node 'App' 'starting' on error { return; }; +show message 'saved' on error { return; }; +validation feedback $Order/Total message 'must be positive' on error { return; }; +show page Module.Page on error { return; }; +close page on error { return; }; +``` + +**Two limits, both reported rather than silently ignored:** + +- **`on error continue` is rejected by Mendix** (CE6035) on `create`, `change`, + `commit`, `log`, `show page`, `close page`, `show message` and + `validation feedback` — **MDL076**. A custom `{ handler }` is accepted on all + of them; `continue` is fine on `declare`, `set`, `retrieve`, `delete` and + `call microflow`. Measured on 11.14.0 — note that create-*variable* and + change-*variable* accept `continue` while change-*object* does not. +- **The list-operation and aggregate forms of `set`** (`$x = head($l)`, + `$n = count($l)`) have no error handling in Mendix at all — **MDL077**. + +**End the handler.** A handler body that does not finish with `return` or `throw` +merges back into the main flow, so a variable created *after* the merge point is +out of scope on the error path — CE0108, which Studio Pro reports for the same +model. Ending the handler (as Studio Pro does when you wire it to an end event) +avoids this entirely. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index e896ef91c2..064ad8a52a 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -121,8 +121,29 @@ func init() { "error", "error handling", "on error", "continue", "rollback", "throw", "exception", "try", "catch", }, - Syntax: "COMMIT $Obj ON ERROR CONTINUE;\nCOMMIT $Obj ON ERROR ROLLBACK;\nCOMMIT $Obj ON ERROR { };\nCOMMIT $Obj ON ERROR WITHOUT ROLLBACK { };", - Example: "COMMIT $Order ON ERROR {\n LOG ERROR 'Failed to save order';\n RETURN empty;\n};\n\nCOMMIT $Batch ON ERROR WITHOUT ROLLBACK {\n LOG WARNING 'Batch save failed, continuing';\n};", + Syntax: "COMMIT $Obj ON ERROR CONTINUE;\nCOMMIT $Obj ON ERROR ROLLBACK;\n" + + "COMMIT $Obj ON ERROR { };\nCOMMIT $Obj ON ERROR WITHOUT ROLLBACK { };\n\n" + + "-- The clause goes on the ACTIVITY that may fail. Most statements take it:\n" + + "-- DECLARE, SET, CREATE, CHANGE, COMMIT, DELETE, RETRIEVE, every CALL,\n" + + "-- LOG, SHOW PAGE, CLOSE PAGE, SHOW MESSAGE, VALIDATION FEEDBACK,\n" + + "-- SYNCHRONIZE, DOWNLOAD FILE and the mapping/REST statements.\n" + + "--\n" + + "-- Two limits, both enforced rather than silently ignored:\n" + + "--\n" + + "-- ON ERROR CONTINUE is rejected by Mendix (CE6035) on CREATE, CHANGE,\n" + + "-- COMMIT, LOG, SHOW PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION\n" + + "-- FEEDBACK -> MDL076. A custom handler IS accepted on all of them, and\n" + + "-- CONTINUE is fine on DECLARE, SET, RETRIEVE, DELETE and CALL MICROFLOW.\n" + + "--\n" + + "-- The list-operation and aggregate forms of SET ($x = head($l),\n" + + "-- $n = count($l)) have no error handling in Mendix at all -> MDL077.\n" + + "--\n" + + "-- A handler that does NOT end in RETURN/THROW merges back into the main\n" + + "-- flow, so a variable created after the merge is out of scope on the error\n" + + "-- path (CE0108). End the handler, or expect that.", + Example: "COMMIT $Order ON ERROR {\n LOG ERROR 'Failed to save order';\n RETURN empty;\n};\n\n" + + "COMMIT $Batch ON ERROR WITHOUT ROLLBACK {\n LOG WARNING 'Batch save failed, continuing';\n};\n\n" + + "DECLARE $Name String = 'default' ON ERROR {\n RETURN 'could not initialise';\n};", SeeAlso: []string{"microflow.control-flow"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index d0ba3dcd8f..152a9a517e 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -553,7 +553,7 @@ it is for pages. | Execute DB query | `$Result = execute database query Module.Conn.Query;` | 3-part name; supports DYNAMIC, params, CONNECTION override | | Import mapping | `[$Var =] import from mapping Module.IMM($SourceVar) [all\|first\|limit [offset ]];` | Apply import mapping to string variable. Trailing clause is Studio Pro's Range; omitted = infer from the mapping's root. `first` binds one OBJECT (`limit 1` is a one-element LIST). Mendix rejects `offset` on a non-list mapping (CE6100) | | Export mapping | `$Var = export to mapping Module.EMM($EntityVar);` | Apply export mapping to entity, returns string | -| Error handling | `... on error continue\|rollback\|{ handler };` | Not supported on EXECUTE DATABASE QUERY | +| Error handling | `... on error continue\|rollback\|{ handler }\|without rollback { handler };` | Goes on the activity that may fail — including `declare`, `set`, `change`, `log`, `show page`, `close page`, `show message` and `validation feedback`, which gained it in mendixlabs/mxcli#1078 so a Studio Pro handler survives DESCRIBE. `on error continue` is refused (MDL076) where Mendix raises CE6035: create, change, commit, log, show page, close page, show message, validation feedback — a custom `{ handler }` is accepted on all of them. The list-operation and aggregate forms of `set` have no error handling at all (MDL077). Not supported on EXECUTE DATABASE QUERY. A handler that does not end in `return`/`throw` merges back into the main flow, so a later variable is out of scope on the error path (CE0108) | **Activity defaults.** An omitted modifier always means Mendix's own default, so a bare MDL statement produces the same activity as dragging a fresh one onto the diff --git a/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl b/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl new file mode 100644 index 0000000000..85b844b1b6 --- /dev/null +++ b/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl @@ -0,0 +1,156 @@ +-- ============================================================================ +-- mendixlabs/mxcli#1078: DESCRIBE dropped an activity's error handler +-- ============================================================================ +-- +-- Report: a microflow with one create-variable activity, error handling set to +-- "custom with rollback", and an error handler wired to an end node. mxcli +-- v0.21 / Mendix 11.12.3. `describe microflow` returned: +-- +-- create or modify microflow ExamplesModule.ACT_noerrorhandler () returns String +-- begin @start(-1196, 200) @position(-1086, 200) +-- declare $name String = 'NameValue'; +-- @position(-943, 200) return $name; end; +-- +-- "No error handler! Regenerating looses the error handler." +-- +-- The report is exactly right, and the loss is silent: the output is valid MDL, +-- `mxcli check` passes, and mxbuild is never involved — so a describe→edit→exec +-- cycle deletes the handler from the model with nothing to warn you. +-- +-- --------------------------------------------------------------------------- +-- TWO STACKED DEFECTS, which is why fixing one looked like fixing both +-- --------------------------------------------------------------------------- +-- +-- 1. THE DESCRIBER (both engines). `getActionErrorHandlingType` was a +-- hand-maintained switch that had drifted to 17 of the 38 action types that +-- store ErrorHandlingType. emitActivityStatement walks an activity's error +-- branch only when hasCustomErrorHandler() agrees, so each of the 21 missing +-- types lost its ENTIRE `on error { … }` block — not just the suffix. +-- Create-object and change-object were among them, i.e. most of what a real +-- microflow is made of. Replaced with a reflection lookup, so a new action +-- type is covered the moment it exists. +-- +-- 2. THE LEGACY PARSER (MXCLI_ENGINE=legacy only). Nine parse functions never +-- read ErrorHandlingType off the BSON, so the value was gone before the +-- describer could be asked. Measured: after fix 1, the default engine +-- round-tripped and legacy still dropped the handler. +-- +-- --------------------------------------------------------------------------- +-- AND A THIRD THING, which fixing #1 alone would have made WORSE +-- --------------------------------------------------------------------------- +-- Eight statement forms had no `onErrorClause` in the grammar — including +-- `declare`, the reporter's own activity. With only fix #1, DESCRIBE emitted +-- +-- declare $name String = 'NameValue' on error { … }; +-- +-- which does not parse: +-- line 3:37 mismatched input 'on' expecting ';' +-- +-- That trades a silent drop for a broken script. So `declare`, `set`, `change`, +-- `log`, `show page`, `close page`, `show message` and `validation feedback` +-- now accept the clause, and the round trip is real rather than cosmetic. +-- +-- --------------------------------------------------------------------------- +-- MEASURED on mxbuild 11.14.0 (baseline: the same project before this script) +-- --------------------------------------------------------------------------- +-- All eight statements below, with a TERMINATING handler: 0 new errors. +-- No CE6035 anywhere, i.e. Mendix accepts a custom handler on all eight. +-- +-- `on error continue` is a different question and the answer is not uniform — +-- one microflow per row, measured, now enforced by MDL076: +-- +-- create variable ok change object CE6035 +-- change variable ok log message CE6035 +-- show page CE6035 +-- close page CE6035 +-- show message CE6035 +-- validation feedback CE6035 +-- +-- Note create-VARIABLE and change-VARIABLE accept Continue while change-OBJECT +-- does not. No rule of thumb predicts that; it was measured. +-- +-- A NON-terminating handler is a different matter and is NOT an mxcli bug: when +-- the handler body falls through, the branch merges back into the main flow, and +-- a variable defined after the merge point is out of scope on the error path — +-- CE0108, which Studio Pro reports for the same model. The handlers below all +-- terminate, matching the reporter's (his goes to an end node). +-- +-- --------------------------------------------------------------------------- +-- WHAT THIS SCRIPT PROVES, AND WHAT IT DOES NOT +-- --------------------------------------------------------------------------- +-- Run it, then `describe microflow` each one: the handlers come back, and +-- describe→exec→describe is byte-identical (verified: the re-exec reports +-- "Unchanged microflow", so nothing was even written). +-- +-- It exercises the WRITE and READ paths together, which is what the issue is +-- about. It does NOT prove the fix against a STUDIO PRO-authored document — +-- the handler here is one mxcli wrote. The reporter's document is the case that +-- motivated it; the unit tests in mdl/executor/microflow_error_handler_*_test.go +-- carry the controls (revert the lookup, all seven cases fail). +-- +-- Entities assumed: MyFirstModule.Car with a Brand attribute, and a page +-- MyFirstModule.Home_Web. Adjust the names for your project. +-- ============================================================================ + +-- The reporter's exact shape: a create-variable with "custom with rollback" +-- whose handler terminates. Before the fix, DESCRIBE returned this microflow +-- with the `on error` block and the `return` inside it both gone. +create or modify microflow MyFirstModule.ACT_1078_Declare () returns String +begin + declare $name String = 'NameValue' on error { + return 'declare failed'; + }; + return $name; +end; + +-- All eight newly-authorable statements in one flow. Every handler terminates, +-- so the whole microflow builds at 0 errors. +create or modify microflow MyFirstModule.ACT_1078_All () returns String +begin + declare $name String = 'NameValue' on error { + return 'declare failed'; + }; + $name = 'changed' on error { + return 'set failed'; + }; + $Car = create MyFirstModule.Car (Brand = 'Ford') on error { + return 'create failed'; + }; + change $Car (Brand = 'Opel') on error { + return 'change failed'; + }; + log info node 'Bug1078' 'hello' on error { + return 'log failed'; + }; + validation feedback $Car/Brand message 'bad brand' on error { + return 'validation failed'; + }; + show message 'hi' on error { + return 'message failed'; + }; + close page on error { + return 'close failed'; + }; + return $name; +end; + +-- `on error without rollback` must survive as its own value, not collapse into +-- the plain custom handler — the two differ at runtime. +create or modify microflow MyFirstModule.ACT_1078_WithoutRollback () returns String +begin + declare $name String = 'NameValue' on error without rollback { + return 'declare failed'; + }; + return $name; +end; + +-- Control for the whole file. No clause is written here, so DESCRIBE must emit +-- none: an invented `on error rollback` is #840, the same bug pointing the other +-- way. Rollback is what an activity with no authored clause stores, so it can +-- never be rendered back. +create or modify microflow MyFirstModule.ACT_1078_NoHandler () returns String +begin + declare $name String = 'NameValue'; + log info node 'Bug1078' 'no handler here'; + return $name; +end; diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index dedcf2aa50..150e1cb983 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -168,10 +168,11 @@ func (s *DropNanoflowStmt) isStatement() {} // DeclareStmt represents: DECLARE $Var Type = expr type DeclareStmt struct { - Variable string // Variable name (without $ prefix) - Type DataType // Variable type - InitialValue Expression // Optional initial value - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + Variable string // Variable name (without $ prefix) + Type DataType // Variable type + InitialValue Expression // Optional initial value + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *DeclareStmt) isMicroflowStatement() {} @@ -232,9 +233,10 @@ func (s *CastObjectStmt) isMicroflowStatement() {} // MfSetStmt represents: SET $Var = expr or SET $Var/Attr = expr // (Named MfSetStmt to avoid conflict with existing SetStmt for SET key = value) type MfSetStmt struct { - Target string // Variable name or attribute path - Value Expression // Value to assign - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + Target string // Variable name or attribute path + Value Expression // Value to assign + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *MfSetStmt) isMicroflowStatement() {} @@ -394,6 +396,7 @@ type ChangeObjectStmt struct { Commit CommitFlag // Commit setting (default CommitNo) RefreshInClient bool // Whether to refresh in client Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *ChangeObjectStmt) isMicroflowStatement() {} @@ -534,11 +537,12 @@ func (p *TemplateParam) IsDataSourceRef() bool { // LogStmt represents: LOG LEVEL [NODE expr] message [WITH params] type LogStmt struct { - Level LogLevel // Log level (INFO, WARNING, etc.) - Node Expression // Optional log node expression - Message Expression // Message expression - Template []TemplateParam // Optional WITH template params - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + Level LogLevel // Log level (INFO, WARNING, etc.) + Node Expression // Optional log node expression + Message Expression // Message expression + Template []TemplateParam // Optional WITH template params + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *LogStmt) isMicroflowStatement() {} @@ -725,6 +729,10 @@ type ListOperationStmt struct { OffsetExpr Expression // Offset expression for RANGE LimitExpr Expression // Limit expression for RANGE Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + // ErrorHandling is recorded only so the clause can be REFUSED. Mendix's + // ListOperationsAction has no ErrorHandlingType, so an ON ERROR here has + // nowhere to go; parsing it and reporting it beats dropping it silently. + ErrorHandling *ErrorHandlingClause } func (s *ListOperationStmt) isMicroflowStatement() {} @@ -785,6 +793,9 @@ type AggregateListStmt struct { ReturnType *DataType Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + // ErrorHandling is recorded only so the clause can be REFUSED — Mendix's + // AggregateAction has no ErrorHandlingType. See ListOperationStmt. + ErrorHandling *ErrorHandlingClause } func (s *AggregateListStmt) isMicroflowStatement() {} @@ -829,13 +840,14 @@ type ShowPageArg struct { // ShowPageStmt represents: SHOW PAGE Module.Page($param = $value) [FOR $obj] [WITH (settings)] type ShowPageStmt struct { - PageName QualifiedName // Page to show - Arguments []ShowPageArg // Page parameter arguments - ForObject string // Optional FOR variable (without $ prefix) - Title string // Optional title override - Location string // Optional location: Content, Popup, Modal (default: Content) - ModalForm bool // Whether to show as modal - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + PageName QualifiedName // Page to show + Arguments []ShowPageArg // Page parameter arguments + ForObject string // Optional FOR variable (without $ prefix) + Title string // Optional title override + Location string // Optional location: Content, Popup, Modal (default: Content) + ModalForm bool // Whether to show as modal + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *ShowPageStmt) isMicroflowStatement() {} @@ -844,6 +856,7 @@ func (s *ShowPageStmt) isMicroflowStatement() {} type ClosePageStmt struct { NumberOfPages int // Number of pages to close (default 1) Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *ClosePageStmt) isMicroflowStatement() {} @@ -857,10 +870,11 @@ func (s *ShowHomePageStmt) isMicroflowStatement() {} // ShowMessageStmt represents: SHOW MESSAGE 'text' TYPE Information OBJECTS [$Var1, $Var2]; type ShowMessageStmt struct { - Message Expression // The message text (string template) - Type string // Information, Warning, Error (default: Information) - TemplateArgs []Expression // Template arguments for message placeholders {1}, {2}, etc. - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + Message Expression // The message text (string template) + Type string // Information, Warning, Error (default: Information) + TemplateArgs []Expression // Template arguments for message placeholders {1}, {2}, etc. + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *ShowMessageStmt) isMicroflowStatement() {} @@ -895,6 +909,7 @@ type ValidationFeedbackStmt struct { Message Expression // The feedback message (string template) TemplateArgs []Expression // Template arguments for message placeholders Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } func (s *ValidationFeedbackStmt) isMicroflowStatement() {} diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index e91c6b5c65..1dfb0b6f30 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -31,9 +31,11 @@ func (fb *flowBuilder) addCreateVariableAction(s *ast.DeclareStmt) model.ID { typeName := declType.Kind.String() fb.declaredVars[s.Variable] = typeName + activityX := fb.posX + action := µflows.CreateVariableAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), VariableName: s.Variable, DataType: convertASTToMicroflowDataType(declType, nil), InitialValue: fb.exprToString(s.InitialValue), @@ -47,12 +49,16 @@ func (fb *flowBuilder) addCreateVariableAction(s *ast.DeclareStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, s.Variable) + return activity.ID } @@ -65,9 +71,11 @@ func (fb *flowBuilder) addChangeVariableAction(s *ast.MfSetStmt) model.ID { errorExampleDeclareVariable(s.Target)) } + activityX := fb.posX + action := µflows.ChangeVariableAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), VariableName: s.Target, Value: fb.exprToString(s.Value), } @@ -80,12 +88,16 @@ func (fb *flowBuilder) addChangeVariableAction(s *ast.MfSetStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, s.Target) + return activity.ID } @@ -316,9 +328,11 @@ func (fb *flowBuilder) addChangeObjectAction(s *ast.ChangeObjectStmt) model.ID { // exec of such actions stays valid without requiring authored MDL to say // `refresh` explicitly; when the author wrote `refresh`, we keep the // same flag for non-empty changes too. + activityX := fb.posX + action := µflows.ChangeObjectAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), ChangeVariable: s.Variable, Commit: commitTypeOf(s.Commit), RefreshInClient: s.RefreshInClient || len(s.Changes) == 0, @@ -349,12 +363,16 @@ func (fb *flowBuilder) addChangeObjectAction(s *ast.ChangeObjectStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, s.Variable) + return activity.ID } diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index b5d452f384..daffbeb5a7 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -77,9 +77,11 @@ func (fb *flowBuilder) addLogMessageAction(s *ast.LogStmt) model.ID { logNodeName = fb.exprToString(s.Node) } + activityX := fb.posX + action := µflows.LogMessageAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), LogLevel: logLevel, LogNodeName: logNodeName, MessageTemplate: &model.Text{ @@ -99,12 +101,16 @@ func (fb *flowBuilder) addLogMessageAction(s *ast.LogStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + return activity.ID } @@ -945,9 +951,11 @@ func (fb *flowBuilder) addShowPageAction(s *ast.ShowPageStmt) model.ID { // Create the action // Use PageName (BY_NAME_REFERENCE) instead of PageID (BY_ID_REFERENCE) // The modern Mendix format uses FormSettings.Form as a qualified name string + activityX := fb.posX + action := µflows.ShowPageAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), PageName: pageQN, // BY_NAME_REFERENCE - qualified name string PageSettings: pageSettings, PageParameterMappings: mappings, @@ -977,12 +985,16 @@ func (fb *flowBuilder) addShowPageAction(s *ast.ShowPageStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + return activity.ID } @@ -1039,9 +1051,11 @@ func (fb *flowBuilder) addShowMessageAction(s *ast.ShowMessageStmt) model.ID { msgType = microflows.MessageTypeInformation } + activityX := fb.posX + action := µflows.ShowMessageAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), Template: template, Type: msgType, TemplateParameters: templateParams, @@ -1055,12 +1069,16 @@ func (fb *flowBuilder) addShowMessageAction(s *ast.ShowMessageStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + return activity.ID } @@ -1138,9 +1156,11 @@ func (fb *flowBuilder) addClosePageAction(s *ast.ClosePageStmt) model.ID { numPages = 1 } + activityX := fb.posX + action := µflows.ClosePageAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), NumberOfPages: numPages, } @@ -1152,12 +1172,16 @@ func (fb *flowBuilder) addClosePageAction(s *ast.ClosePageStmt) model.ID { Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + return activity.ID } @@ -1254,9 +1278,11 @@ func (fb *flowBuilder) addValidationFeedbackAction(s *ast.ValidationFeedbackStmt varName = varName[1:] } + activityX := fb.posX + action := µflows.ValidationFeedbackAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: fb.ehType(nil), + ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), ObjectVariable: varName, AttributeName: attributeName, AssociationName: associationName, @@ -1272,12 +1298,16 @@ func (fb *flowBuilder) addValidationFeedbackAction(s *ast.ValidationFeedbackStmt Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, }, AutoGenerateCaption: true, + ErrorHandlingType: fb.ehType(s.ErrorHandling), }, Action: action, } fb.objects = append(fb.objects, activity) fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + return activity.ID } diff --git a/mdl/executor/cmd_microflows_show_helpers.go b/mdl/executor/cmd_microflows_show_helpers.go index 736984463e..5995e27b62 100644 --- a/mdl/executor/cmd_microflows_show_helpers.go +++ b/mdl/executor/cmd_microflows_show_helpers.go @@ -6,6 +6,7 @@ package executor import ( "context" "fmt" + "reflect" "sort" "strconv" "strings" @@ -2040,53 +2041,59 @@ func hasCustomErrorHandler(errType microflows.ErrorHandlingType) bool { // getActionErrorHandlingType extracts the ErrorHandlingType from the action inside an ActionActivity. // Most action types store ErrorHandlingType at the action level, not the activity level. +// +// This gates far more than a suffix. emitActivityStatement only walks an +// activity's error branch when hasCustomErrorHandler() agrees, so an action whose +// type is not reported here loses its ENTIRE `on error { … }` block from DESCRIBE +// — silently, and in valid-looking MDL, so a describe→edit→exec round-trip +// deletes the handler from the model (mendixlabs/mxcli#1078). func getActionErrorHandlingType(activity *microflows.ActionActivity) microflows.ErrorHandlingType { if activity == nil || activity.Action == nil { return "" } - switch action := activity.Action.(type) { - case *microflows.MicroflowCallAction: - return action.ErrorHandlingType - case *microflows.NanoflowCallAction: - return action.ErrorHandlingType - case *microflows.JavaActionCallAction: - return action.ErrorHandlingType - case *microflows.JavaScriptActionCallAction: - return action.ErrorHandlingType - case *microflows.CallExternalAction: - return action.ErrorHandlingType - case *microflows.RestCallAction: - return action.ErrorHandlingType - case *microflows.WebServiceCallAction: - return action.ErrorHandlingType - case *microflows.RestOperationCallAction: - return "" // RestOperationCallAction does not support custom error handling (CE6035) - case *microflows.ExecuteDatabaseQueryAction: - return action.ErrorHandlingType - case *microflows.ImportXmlAction: - return action.ErrorHandlingType - case *microflows.ExportXmlAction: - return action.ErrorHandlingType - case *microflows.CommitObjectsAction: - return action.ErrorHandlingType - case *microflows.RetrieveAction: - return action.ErrorHandlingType - case *microflows.DeleteObjectAction: - return action.ErrorHandlingType - case *microflows.DownloadFileAction: - return action.ErrorHandlingType - case *microflows.SynchronizeAction: - return action.ErrorHandlingType - case *microflows.UnsupportedAction: - // Read off the stored action by property name — see errorHandlingTypeOf. - // Without this the handler on an unmapped action reads as "no error - // handling" and its branch is dropped (#863). - return action.ErrorHandlingType - default: - // Fall back to activity level for action types without ErrorHandlingType field - return activity.ErrorHandlingType + // The one deliberate exclusion: the field is stored, but Mendix refuses a + // custom handler on this action (CE6035), so reporting it would render a + // block that cannot be executed back. + if _, ok := activity.Action.(*microflows.RestOperationCallAction); ok { + return "" + } + + if errType := actionErrorHandlingField(activity.Action); errType != "" { + return errType + } + // Fall back to activity level for action types without ErrorHandlingType field. + return activity.ErrorHandlingType +} + +// actionErrorHandlingField reads ErrorHandlingType off any action that declares it. +// +// Reflection rather than a case per action type. The hand-maintained switch this +// replaces had drifted to 17 of the 38 action types that carry the field, and +// every one of the 21 it missed — CreateObjectAction and ChangeObjectAction +// among them — dropped that activity's whole error branch from DESCRIBE. The +// list had already been patched twice for individual instances (#863, #1020's +// neighbour) and silently regrew, which is what makes an enumeration the wrong +// shape here: a new action type must not have to be remembered. +// +// Actions embed model.BaseElement, which has no such field, so a promoted field +// cannot be picked up by accident. +func actionErrorHandlingField(action microflows.MicroflowAction) microflows.ErrorHandlingType { + v := reflect.ValueOf(action) + for v.Kind() == reflect.Ptr { + if v.IsNil() { + return "" + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return "" + } + f := v.FieldByName("ErrorHandlingType") + if !f.IsValid() || f.Kind() != reflect.String { + return "" } + return microflows.ErrorHandlingType(f.String()) } // collectErrorHandlerStatements traverses the error handler flow and collects statements. diff --git a/mdl/executor/microflow_error_handler_authoring_test.go b/mdl/executor/microflow_error_handler_authoring_test.go new file mode 100644 index 0000000000..c57a2b2c6b --- /dev/null +++ b/mdl/executor/microflow_error_handler_authoring_test.go @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/exprcheck" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1078, second half. Fixing the describer alone made it emit an +// `on error { … }` block on eight statements whose grammar had no onErrorClause +// — MDL that does not parse. Trading a silent drop for a broken script is not a +// fix, so those eight statements now accept the clause. +// +// The reporter's activity is the first row: a create-variable with "custom with +// rollback", which is ErrorHandlingType "Custom" in the metamodel. + +// buildFlowFromMDL parses a microflow and returns the objects the flow builder +// produced. Parsed rather than hand-built: the point is that the CLAUSE reaches +// the model from real source text, so a hand-made AST would test nothing. +func buildFlowFromMDL(t *testing.T, body string) *flowBuilder { + t.Helper() + prog, errs := visitor.Build("create microflow M.ACT_T()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatalf("parsing:\n%s\nerrors: %v", body, errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + fb := &flowBuilder{ + posX: 100, posY: 100, spacing: HorizontalSpacing, + varTypes: map[string]string{}, declaredVars: map[string]string{}, + } + fb.buildFlowGraph(mf.Body, nil) + return fb +} + +// actionErrorHandlingTypes returns every action activity's ErrorHandlingType. +// All of them, not the first: a handler body contributes activities too, and the +// `set` case needs a declare in front of it. +func actionErrorHandlingTypes(fb *flowBuilder) []microflows.ErrorHandlingType { + var out []microflows.ErrorHandlingType + for _, o := range fb.objects { + if a, ok := o.(*microflows.ActionActivity); ok { + out = append(out, actionErrorHandlingField(a.Action)) + } + } + return out +} + +// countErrorHandling returns how many action activities carry the given type. +func countErrorHandling(fb *flowBuilder, want microflows.ErrorHandlingType) int { + var n int + for _, t := range actionErrorHandlingTypes(fb) { + if t == want { + n++ + } + } + return n +} + +// Each of the eight statements #1078 opened up, with the reporter's own shape +// first. Before the grammar change every one of these was a parse error. +func TestAuthorOnError_EightStatementsThatCouldNotCarryTheClause(t *testing.T) { + for _, tc := range []struct { + name string + body string + }{ + {"declare", "declare $name String = 'NameValue' on error { rollback $x; };"}, + {"set", "declare $name String = 'v';\n$name = 'w' on error { rollback $x; };"}, + {"change object", "change $Car (Brand = 'Opel') on error { rollback $x; };"}, + {"log", "log info node 'B' 'hi' on error { rollback $x; };"}, + {"show page", "show page M.Home on error { rollback $x; };"}, + {"close page", "close page on error { rollback $x; };"}, + {"show message", "show message 'hi' on error { rollback $x; };"}, + {"validation feedback", "validation feedback $Car/Brand message 'bad' on error { rollback $x; };"}, + } { + t.Run(tc.name, func(t *testing.T) { + fb := buildFlowFromMDL(t, tc.body) + + if len(actionErrorHandlingTypes(fb)) == 0 { + t.Fatal("the builder produced no action activity") + } + // Custom is what Studio Pro's "custom with rollback" stores, and it is + // what hasCustomErrorHandler needs to see for DESCRIBE to walk the + // branch back out again. Exactly one activity carries it: the one the + // clause was written on. + if n := countErrorHandling(fb, microflows.ErrorHandlingTypeCustom); n != 1 { + t.Errorf("%d activities carry ErrorHandlingType %q, want 1 — DESCRIBE "+ + "gates the whole branch on this value; got %v", + n, microflows.ErrorHandlingTypeCustom, actionErrorHandlingTypes(fb)) + } + + // The handler's own activities and the error flow must exist, or the + // clause parsed into nothing. + var errFlows int + for _, f := range fb.flows { + if f.IsErrorHandler { + errFlows++ + } + } + if errFlows != 1 { + t.Errorf("got %d error-handler flows, want 1 — the handler body was not wired", errFlows) + } + }) + } +} + +// Control for the table above. Without the clause the action must carry NO +// error-handling type: writing one would make DESCRIBE render `on error` on an +// activity the author never put one on, which is #840 in reverse. +func TestAuthorOnError_AbsentClauseLeavesTheActionUnset(t *testing.T) { + for _, body := range []string{ + "declare $name String = 'NameValue';", + "log info node 'B' 'hi';", + "close page;", + } { + fb := buildFlowFromMDL(t, body) + types := actionErrorHandlingTypes(fb) + if len(types) == 0 { + t.Fatalf("%s: no action activity", body) + } + for _, errType := range types { + if errType != "" { + t.Errorf("%s: ErrorHandlingType = %q, want empty — no clause was written", + body, errType) + } + } + for _, f := range fb.flows { + if f.IsErrorHandler { + t.Errorf("%s: an error-handler flow was created with no clause", body) + } + } + } +} + +// `on error without rollback` must reach the model as its own value, not collapse +// into Custom — the two differ at runtime. +func TestAuthorOnError_WithoutRollbackIsDistinct(t *testing.T) { + fb := buildFlowFromMDL(t, + "declare $name String = 'v' on error without rollback { rollback $x; };") + if n := countErrorHandling(fb, microflows.ErrorHandlingTypeCustomWithoutRollback); n != 1 { + t.Errorf("%d activities carry %q, want 1; got %v", n, + microflows.ErrorHandlingTypeCustomWithoutRollback, actionErrorHandlingTypes(fb)) + } +} + +// MDL077: `set` is one statement form spanning activities that can and cannot +// hold the clause. Change variable can; list operation and aggregate have no +// ErrorHandlingType in the metamodel at all, so the clause is refused rather than +// accepted and discarded. +func TestMDL077_RefusesOnErrorWhereTheActivityHasNoField(t *testing.T) { + refused := []struct{ name, body string }{ + {"list operation", "$h = head($list) on error { rollback $x; };"}, + {"aggregate", "$n = count($list) on error { rollback $x; };"}, + } + for _, tc := range refused { + t.Run(tc.name, func(t *testing.T) { + out := checkMicroflowBodyForTest(t, tc.body) + if !strings.Contains(out, "MDL077") { + t.Errorf("expected MDL077 for %s, got:\n%s", tc.name, out) + } + }) + } + + // Control: the plain form of the SAME statement keyword is accepted, so the + // rule is discriminating between activities and not just banning `set`. + if out := checkMicroflowBodyForTest(t, + "declare $a String = 'v';\n$a = 'w' on error { rollback $x; };"); strings.Contains(out, "MDL077") { + t.Errorf("MDL077 fired on a plain set, which IS a Change variable activity:\n%s", out) + } +} + +// checkMicroflowBodyForTest runs the microflow validator over a body and returns +// the rule IDs it reported. +func checkMicroflowBodyForTest(t *testing.T, body string) string { + t.Helper() + prog, errs := visitor.Build("create microflow M.ACT_T()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatalf("parsing:\n%s\nerrors: %v", body, errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + v := µflowValidator{ + mfName: "M.ACT_T", + emptyListVars: map[string]bool{}, + varKinds: map[string]exprcheck.TypeKind{}, + } + v.walkBody(mf.Body) + var b strings.Builder + for _, viol := range v.violations { + b.WriteString(viol.RuleID + ": " + viol.Message + "\n") + } + return b.String() +} diff --git a/mdl/executor/microflow_error_handler_roundtrip_test.go b/mdl/executor/microflow_error_handler_roundtrip_test.go new file mode 100644 index 0000000000..f4b9dec01b --- /dev/null +++ b/mdl/executor/microflow_error_handler_roundtrip_test.go @@ -0,0 +1,198 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1078: a microflow whose activity carried a custom error +// handler came back from DESCRIBE with the handler — and the whole branch behind +// it — gone, so a describe→edit→exec round-trip deleted it from the model. +// +// The gate is getActionErrorHandlingType. emitActivityStatement only walks the +// error branch when hasCustomErrorHandler() agrees, and the hand-maintained +// switch that fed it covered 17 of the 38 action types that store the field. The +// reported activity (create variable) was one of the 21 it missed; so were +// create-object and change-object, which is most of what a real microflow is +// made of. +// +// The output stayed valid MDL throughout, which is why nothing caught it: no +// parse error, no mx check finding, just a smaller microflow. + +// errorHandlerCase is one action type that had no case in the old switch. +type errorHandlerCase struct { + name string + action microflows.MicroflowAction + want string // a fragment of the statement DESCRIBE should render +} + +// The action types the switch missed, one per family. Every one of these stores +// ErrorHandlingType and every one lost its branch before the fix. +func missingErrorHandlerCases() []errorHandlerCase { + return []errorHandlerCase{ + {"create variable", µflows.CreateVariableAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + VariableName: "name", DataType: µflows.StringType{}, + InitialValue: "'NameValue'", + }, "declare $name"}, + {"change variable", µflows.ChangeVariableAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + VariableName: "name", Value: "'other'", + }, "set $name"}, + {"create object", µflows.CreateObjectAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + EntityQualifiedName: "Mod.Car", OutputVariable: "Car", + }, "create Mod.Car"}, + {"change object", µflows.ChangeObjectAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + ChangeVariable: "Car", + }, "change $Car"}, + {"log message", µflows.LogMessageAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + }, "log "}, + {"close page", µflows.ClosePageAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, NumberOfPages: 1, + }, "close page"}, + {"custom without rollback", µflows.CreateVariableAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustomWithoutRollback, + VariableName: "n", DataType: µflows.StringType{}, InitialValue: "'v'", + }, "declare $n"}, + } +} + +// describeWithErrorHandler runs the real traversal over a two-activity flow whose +// first activity carries a custom handler, and returns the rendered MDL. +func describeWithErrorHandler(t *testing.T, action microflows.MicroflowAction) string { + t.Helper() + e := newTestExecutor() + + activityMap := map[model.ID]microflows.MicroflowObject{ + mkID("start"): µflows.StartEvent{BaseMicroflowObject: mkObj("start")}, + mkID("act"): µflows.ActionActivity{ + BaseActivity: microflows.BaseActivity{BaseMicroflowObject: mkObj("act")}, + Action: action, + }, + // The error branch. Its statement is the canary: if the branch is not + // traversed, this string is absent from the output entirely. + mkID("err"): µflows.ActionActivity{ + BaseActivity: microflows.BaseActivity{BaseMicroflowObject: mkObj("err")}, + Action: µflows.RollbackObjectAction{RollbackVariable: "Handled"}, + }, + mkID("end"): µflows.EndEvent{BaseMicroflowObject: mkObj("end")}, + } + flowsByOrigin := map[model.ID][]*microflows.SequenceFlow{ + mkID("start"): {mkFlow("start", "act")}, + mkID("act"): {mkFlow("act", "end"), mkErrorFlow("act", "err")}, + } + + var lines []string + e.traverseFlow(mkID("start"), activityMap, flowsByOrigin, nil, + make(map[model.ID]bool), nil, nil, &lines, 1, nil, 0, nil) + return strings.Join(lines, "\n") +} + +// The regression itself. With the reflection lookup reverted to the old switch, +// every subtest here fails on the first assertion — the branch statement is +// simply not in the output. +func TestDescribe_ErrorHandlerSurvives_ActionsTheSwitchMissed(t *testing.T) { + for _, tc := range missingErrorHandlerCases() { + t.Run(tc.name, func(t *testing.T) { + out := describeWithErrorHandler(t, tc.action) + + if !strings.Contains(out, "rollback $Handled;") { + t.Errorf("the error branch was dropped — a describe→exec round-trip "+ + "would delete it from the model (#1078):\n%s", out) + } + if !strings.Contains(out, "on error") { + t.Errorf("output does not mark the handler, so the branch would be "+ + "re-executed unconditionally in the main flow:\n%s", out) + } + // Live MDL, not the commented-out fallback: these statements can all + // carry an onErrorClause, so the round trip must actually round-trip. + if strings.Contains(out, "-- on error") { + t.Errorf("handler rendered commented-out, but this statement can carry "+ + "the clause — describe→exec would still lose it:\n%s", out) + } + if !strings.Contains(out, tc.want) { + t.Errorf("statement %q missing from output:\n%s", tc.want, out) + } + }) + } +} + +// Control for the test above. Without this, a "fix" that reported a custom +// handler for EVERY action would pass every case in the table while inventing +// handlers on activities that have none — and DESCRIBE would grow an `on error` +// block on activities the author never put one on. +func TestDescribe_NoErrorHandler_WhenTheActionHasNone(t *testing.T) { + // Same flow shape, same error flow present in the model, but the action + // carries no error-handling type at all. + out := describeWithErrorHandler(t, µflows.CreateVariableAction{ + VariableName: "name", DataType: µflows.StringType{}, InitialValue: "'v'", + }) + if strings.Contains(out, "on error {") { + t.Errorf("a live `on error { }` block was rendered for an action with no "+ + "error handling type:\n%s", out) + } +} + +// Rollback is deliberately excluded and must stay excluded: it is what an +// activity with NO authored clause stores, so rendering it would put a clause in +// the user's script that they never wrote (#840). Keeping this beside the #1078 +// cases pins both directions of the same decision. +func TestDescribe_RollbackIsStillNotRendered(t *testing.T) { + out := describeWithErrorHandler(t, µflows.CreateVariableAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeRollback, + VariableName: "name", DataType: µflows.StringType{}, InitialValue: "'v'", + }) + if strings.Contains(out, "on error rollback") { + t.Errorf("#840 regression: `on error rollback` rendered for the default:\n%s", out) + } +} + +// actionErrorHandlingField is the reflection lookup that replaced the switch. +// Pinning it directly is what makes the fix durable: a NEW action type carrying +// ErrorHandlingType is handled the moment it exists, with nothing to remember. +func TestActionErrorHandlingField(t *testing.T) { + for _, tc := range []struct { + name string + action microflows.MicroflowAction + want microflows.ErrorHandlingType + }{ + {"reads the field", µflows.CreateVariableAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom}, microflows.ErrorHandlingTypeCustom}, + {"empty when unset", µflows.CreateVariableAction{}, ""}, + {"action without the field", µflows.ListOperationAction{}, ""}, + {"nil action", nil, ""}, + } { + if got := actionErrorHandlingField(tc.action); got != tc.want { + t.Errorf("%s: got %q, want %q", tc.name, got, tc.want) + } + } + + // A typed-nil pointer must not panic — activity.Action can hold one. + var typedNil *microflows.CreateVariableAction + if got := actionErrorHandlingField(typedNil); got != "" { + t.Errorf("typed nil: got %q, want empty", got) + } +} + +// RestOperationCallAction stores the field but Mendix refuses a custom handler on +// it (CE6035), so it is the one action deliberately not reported. A reflection +// lookup would otherwise pick it up — this is the case that stops the generic fix +// from being too generic. +func TestGetActionErrorHandlingType_RestOperationCallIsExcluded(t *testing.T) { + activity := µflows.ActionActivity{ + Action: µflows.RestOperationCallAction{ + ErrorHandlingType: microflows.ErrorHandlingTypeCustom, + }, + } + if got := getActionErrorHandlingType(activity); got != "" { + t.Errorf("got %q, want empty — Mendix rejects a custom handler here with CE6035", got) + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index 0d0ef4626f..cb00ce949a 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -195,6 +195,7 @@ func (v *microflowValidator) walkBody(body []ast.MicroflowStatement) { for _, s := range body { v.checkUnknownAnnotations(s) v.checkErrorHandlingContinueSupported(s) + v.checkErrorHandlingSupported(s) switch stmt := s.(type) { case *ast.ValidationFeedbackStmt: if isEmptyMessage(stmt.Message) { @@ -1343,6 +1344,29 @@ func stmtErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClause { return s.ErrorHandling case *ast.ExecuteDatabaseQueryStmt: return s.ErrorHandling + // The eight statements #1078 gave an onErrorClause. Without them here, MDL076 + // cannot see a clause these statements now accept, and MDL077 cannot refuse + // one on a list operation or aggregate. + case *ast.DeclareStmt: + return s.ErrorHandling + case *ast.MfSetStmt: + return s.ErrorHandling + case *ast.ChangeObjectStmt: + return s.ErrorHandling + case *ast.LogStmt: + return s.ErrorHandling + case *ast.ShowPageStmt: + return s.ErrorHandling + case *ast.ClosePageStmt: + return s.ErrorHandling + case *ast.ShowMessageStmt: + return s.ErrorHandling + case *ast.ValidationFeedbackStmt: + return s.ErrorHandling + case *ast.ListOperationStmt: + return s.ErrorHandling + case *ast.AggregateListStmt: + return s.ErrorHandling } return nil } diff --git a/mdl/executor/validate_microflow_error_handling.go b/mdl/executor/validate_microflow_error_handling.go index c0fc43ba93..5a59b86855 100644 --- a/mdl/executor/validate_microflow_error_handling.go +++ b/mdl/executor/validate_microflow_error_handling.go @@ -25,11 +25,17 @@ const continueUnsupportedRule = "MDL076" // Retrieve ok ok ok // Delete ok ok ok // MicroflowCall ok ok CE6035 +// CreateVariable ok ok - +// ChangeVariable ok ok - // Log ok CE6035 CE6035 // Create ok CE6035 CE6035 // Change ok CE6035 CE6035 // Commit ok CE6035 CE6035 // Aggregate ok CE6035 CE6035 +// ShowPage ok CE6035 - +// ClosePage ok CE6035 - +// ShowMessage ok CE6035 - +// ValidationFeedback ok CE6035 - // // A DENY-list rather than an allow-list, deliberately. The statements not named // here were never measured, and refusing them would reject scripts that may well @@ -37,14 +43,21 @@ const continueUnsupportedRule = "MDL076" // That is also why the table is per-statement and not "activities that write to // the database": Delete writes and is fine, Aggregate does not and is not. // -// Only CREATE and COMMIT appear, because they are the only rejecting activities -// MDL can even put the clause on: `logStatement` and the change statement carry -// no onErrorClause in the grammar, so Log and Change are unreachable from a -// script however badly they behave in a stored document. Adding them would be -// dead code that reads as coverage. +// The bottom nine rows became REACHABLE with mendixlabs/mxcli#1078, which gave +// eight more statements an onErrorClause so a Studio Pro error handler could +// survive DESCRIBE. Before that they were unreachable from a script and this list +// held only create and commit. Note the split it exposes, which no rule of thumb +// predicts: create-VARIABLE and change-VARIABLE accept Continue while +// change-OBJECT does not — measured on 11.14.0, one microflow per row. var continueUnsupportedOn = map[string]string{ - "create": "Create object activity", - "commit": "Commit object(s) activity", + "create": "Create object activity", + "commit": "Commit object(s) activity", + "change": "Change object activity", + "log": "Log message activity", + "show page": "Show page activity", + "close page": "Close page activity", + "show message": "Show message activity", + "validation feedback": "Validation feedback activity", } // checkErrorHandlingContinueSupported reports `ON ERROR CONTINUE` on a statement @@ -89,8 +102,61 @@ func continueUnsupportedStatement(stmt ast.MicroflowStatement) (keyword, activit keyword = "create" case *ast.MfCommitStmt: keyword = "commit" + case *ast.ChangeObjectStmt: + keyword = "change" + case *ast.LogStmt: + keyword = "log" + case *ast.ShowPageStmt: + keyword = "show page" + case *ast.ClosePageStmt: + keyword = "close page" + case *ast.ShowMessageStmt: + keyword = "show message" + case *ast.ValidationFeedbackStmt: + keyword = "validation feedback" default: + // DeclareStmt and MfSetStmt are deliberately absent: create-variable and + // change-variable accept Continue on 11.14.0. return "", "" } return keyword, continueUnsupportedOn[keyword] } + +// errorHandlingUnavailableRule flags an ON ERROR clause on a statement whose +// Mendix activity has no ErrorHandlingType property at all. +const errorHandlingUnavailableRule = "MDL077" + +// checkErrorHandlingSupported reports an ON ERROR clause the stored activity +// cannot hold. +// +// The `set` statement is overloaded: `$x = $y` is a Change variable activity, +// which HAS an ErrorHandlingType, while `$x = head($list)` and `$x = count($list)` +// are List operation and Aggregate activities, which do not — the property is +// absent from Microflows$ListOperationsAction and Microflows$AggregateAction in +// the metamodel, not merely unset. One MDL statement form therefore spans +// activities that can and cannot carry the clause. +// +// Refused rather than dropped. #1078 exists because an error handler that +// disappears between the model and the script is invisible until someone +// re-executes the script and finds the handler gone; accepting a clause here and +// writing nothing would rebuild that same trap one statement over. +func (v *microflowValidator) checkErrorHandlingSupported(stmt ast.MicroflowStatement) { + if stmtErrorHandling(stmt) == nil { + return + } + var form, activity string + switch stmt.(type) { + case *ast.ListOperationStmt: + form, activity = "a list operation", "List operation activity" + case *ast.AggregateListStmt: + form, activity = "an aggregate", "Aggregate list activity" + default: + return + } + v.addViolation(errorHandlingUnavailableRule, linter.SeverityError, + fmt.Sprintf("`on error` is not available on %s — Mendix's %s has no error-handling "+ + "property, so the clause could only be discarded", form, activity), + "Drop the clause. To handle a failure around it, put the statement inside the "+ + "custom handler of an activity that does support one, or split the expression "+ + "out into a plain `set` (a Change variable activity), which does.") +} diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 067aca6d59..8e2be61c46 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -249,7 +249,7 @@ microflowStatement ; declareStatement - : DECLARE VARIABLE dataType (EQUALS expression)? + : DECLARE VARIABLE dataType (EQUALS expression)? onErrorClause? ; caseStatement @@ -304,7 +304,7 @@ castObjectStatement // rule unguessable — and the parse error named the token, not the missing // keyword (mxcli-formula1 findings #13). setStatement - : SET? (VARIABLE | attributePath) EQUALS expression + : SET? (VARIABLE | attributePath) EQUALS expression onErrorClause? ; // $NewProduct = CREATE MfTest.Product (Name = $Name, Code = $Code); @@ -317,7 +317,7 @@ createObjectStatement // CHANGE $Product (Name = $NewName, ModifiedDate = [%CurrentDateTime%]); // CHANGE $Product (Name = $NewName) COMMIT WITHOUT EVENTS REFRESH; changeObjectStatement - : CHANGE VARIABLE (LPAREN memberAssignmentList? RPAREN)? commitClause? REFRESH? + : CHANGE VARIABLE (LPAREN memberAssignmentList? RPAREN)? commitClause? REFRESH? onErrorClause? ; // The Commit flag on a create/change activity: Mendix's Microflows$Commit enum. @@ -421,7 +421,7 @@ raiseErrorStatement // LOG INFO NODE 'TEST' 'Message'; or LOG INFO 'Message'; or LOG WARNING 'Message' WITH ({1} = $var); logStatement - : LOG logLevel? (NODE expression)? expression logTemplateParams? + : LOG logLevel? (NODE expression)? expression logTemplateParams? onErrorClause? ; logLevel @@ -587,7 +587,7 @@ callArgument ; showPageStatement - : SHOW PAGE qualifiedName (LPAREN showPageArgList? RPAREN)? (FOR VARIABLE)? (WITH memberAssignmentList)? + : SHOW PAGE qualifiedName (LPAREN showPageArgList? RPAREN)? (FOR VARIABLE)? (WITH memberAssignmentList)? onErrorClause? ; showPageArgList @@ -600,7 +600,7 @@ showPageArg ; closePageStatement - : CLOSE PAGE + : CLOSE PAGE onErrorClause? ; showHomePageStatement @@ -609,7 +609,7 @@ showHomePageStatement // SHOW MESSAGE 'Hello {1}' TYPE Information OBJECTS [$Name]; showMessageStatement - : SHOW MESSAGE expression (TYPE identifierOrKeyword)? (OBJECTS LBRACKET expressionList RBRACKET)? + : SHOW MESSAGE expression (TYPE identifierOrKeyword)? (OBJECTS LBRACKET expressionList RBRACKET)? onErrorClause? ; // SYNCHRONIZE ALL; @@ -635,7 +635,7 @@ throwStatement // VALIDATION FEEDBACK $Product/Code MESSAGE 'Product code cannot be empty'; validationFeedbackStatement - : VALIDATION FEEDBACK (attributePath | VARIABLE) MESSAGE expression (OBJECTS LBRACKET expressionList RBRACKET)? + : VALIDATION FEEDBACK (attributePath | VARIABLE) MESSAGE expression (OBJECTS LBRACKET expressionList RBRACKET)? onErrorClause? ; // ============================================================================= diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index 63d81e40a6..c299ed7900 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -66,6 +66,11 @@ func buildLogStatement(ctx parser.ILogStatementContext) *ast.LogStmt { } } + // Check for ON ERROR clause + if errClause := logCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } @@ -1182,6 +1187,11 @@ func buildShowPageStatement(ctx parser.IShowPageStatementContext) *ast.ShowPageS } } + // Check for ON ERROR clause + if errClause := showCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } @@ -1255,6 +1265,11 @@ func buildShowMessageStatement(ctx parser.IShowMessageStatementContext) *ast.Sho } } + // Check for ON ERROR clause + if errClause := smCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } @@ -1338,6 +1353,11 @@ func buildValidationFeedbackStatement(ctx parser.IValidationFeedbackStatementCon } } + // Check for ON ERROR clause + if errClause := vfCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } diff --git a/mdl/visitor/visitor_microflow_statements.go b/mdl/visitor/visitor_microflow_statements.go index 2410499df1..0acaa20149 100644 --- a/mdl/visitor/visitor_microflow_statements.go +++ b/mdl/visitor/visitor_microflow_statements.go @@ -107,8 +107,12 @@ func buildMicroflowStatement(ctx parser.IMicroflowStatementContext) ast.Microflo stmt = buildRemoveFromListStatement(removeFrom) } else if showPage := mfCtx.ShowPageStatement(); showPage != nil { stmt = buildShowPageStatement(showPage) - } else if mfCtx.ClosePageStatement() != nil { - stmt = &ast.ClosePageStmt{NumberOfPages: 1} + } else if closePage := mfCtx.ClosePageStatement(); closePage != nil { + close := &ast.ClosePageStmt{NumberOfPages: 1} + if errClause := closePage.OnErrorClause(); errClause != nil { + close.ErrorHandling = buildOnErrorClause(errClause) + } + stmt = close } else if mfCtx.ShowHomePageStatement() != nil { stmt = &ast.ShowHomePageStmt{} } else if showMsg := mfCtx.ShowMessageStatement(); showMsg != nil { @@ -668,6 +672,11 @@ func buildDeclareStatement(ctx parser.IDeclareStatementContext) *ast.DeclareStmt stmt.InitialValue = appendStatementExpressionTrailingWhitespace(expr, stmt.InitialValue) } + // Check for ON ERROR clause + if errClause := declCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } @@ -725,7 +734,34 @@ func buildCastObjectStatement(ctx parser.ICastObjectStatementContext) *ast.CastO // buildSetStatement converts SET statement context to MfSetStmt or specialized statement types. // When the expression is a list operation (HEAD, TAIL, etc.) or aggregate (COUNT, SUM, etc.), // this returns the specialized statement type instead of MfSetStmt. +// +// The ON ERROR clause is attached here rather than at each of the dozen return +// points below. Only the plain MfSetStmt form can honour it — Mendix gives +// ChangeVariableAction an ErrorHandlingType but gives ListOperationsAction and +// AggregateAction none — so the specialized nodes carry it only to be refused by +// MDL077, never to be executed. func buildSetStatement(ctx parser.ISetStatementContext) ast.MicroflowStatement { + stmt := buildSetStatementNode(ctx) + if stmt == nil || ctx == nil { + return stmt + } + errClause := ctx.(*parser.SetStatementContext).OnErrorClause() + if errClause == nil { + return stmt + } + eh := buildOnErrorClause(errClause) + switch s := stmt.(type) { + case *ast.MfSetStmt: + s.ErrorHandling = eh + case *ast.ListOperationStmt: + s.ErrorHandling = eh + case *ast.AggregateListStmt: + s.ErrorHandling = eh + } + return stmt +} + +func buildSetStatementNode(ctx parser.ISetStatementContext) ast.MicroflowStatement { if ctx == nil { return nil } @@ -1098,6 +1134,11 @@ func buildChangeObjectStatement(ctx parser.IChangeObjectStatementContext) *ast.C stmt.Commit = buildCommitClause(changeCtx.CommitClause()) stmt.RefreshInClient = changeCtx.REFRESH() != nil + // Check for ON ERROR clause + if errClause := changeCtx.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt } diff --git a/sdk/mpr/parser_microflow.go b/sdk/mpr/parser_microflow.go index c6c58ef1e2..9f4fc67533 100644 --- a/sdk/mpr/parser_microflow.go +++ b/sdk/mpr/parser_microflow.go @@ -643,6 +643,9 @@ func parseMicroflowActionValue(raw any) microflows.MicroflowAction { func parseCreateVariableAction(raw map[string]any) *microflows.CreateVariableAction { action := µflows.CreateVariableAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.VariableName = extractString(raw["VariableName"]) action.InitialValue = extractString(raw["InitialValue"]) @@ -656,6 +659,9 @@ func parseCreateVariableAction(raw map[string]any) *microflows.CreateVariableAct func parseChangeVariableAction(raw map[string]any) *microflows.ChangeVariableAction { action := µflows.ChangeVariableAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.VariableName = extractString(raw["ChangeVariableName"]) action.Value = extractString(raw["Value"]) return action @@ -664,6 +670,9 @@ func parseChangeVariableAction(raw map[string]any) *microflows.ChangeVariableAct func parseCreateObjectAction(raw map[string]any) *microflows.CreateObjectAction { action := µflows.CreateObjectAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) // Entity is BY_NAME_REFERENCE - can be string (qualified name) or binary (legacy) if entityStr, ok := raw["Entity"].(string); ok { action.EntityQualifiedName = entityStr @@ -696,6 +705,9 @@ func parseCreateObjectAction(raw map[string]any) *microflows.CreateObjectAction func parseChangeObjectAction(raw map[string]any) *microflows.ChangeObjectAction { action := µflows.ChangeObjectAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.ChangeVariable = extractString(raw["ChangeVariableName"]) action.RefreshInClient = extractBool(raw["RefreshInClient"], false) diff --git a/sdk/mpr/parser_microflow_actions.go b/sdk/mpr/parser_microflow_actions.go index 0f1a89ab48..914a6b74b8 100644 --- a/sdk/mpr/parser_microflow_actions.go +++ b/sdk/mpr/parser_microflow_actions.go @@ -195,6 +195,9 @@ func parseCodeActionParameterValue(raw map[string]any) microflows.CodeActionPara func parseShowPageAction(raw map[string]any) *microflows.ShowPageAction { action := µflows.ShowPageAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.PageID = model.ID(extractBsonID(raw["Page"])) action.PassedObject = extractString(raw["PassedObjectVariableName"]) @@ -294,6 +297,9 @@ func parseShowHomePageAction(raw map[string]any) *microflows.ShowHomePageAction func parseClosePageAction(raw map[string]any) *microflows.ClosePageAction { action := µflows.ClosePageAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) // Issue #585: collapse the int32/int64 dispatch to the shared extractInt // helper. Default of 1 is preserved when the field is absent. // Storage name is "NumberOfPages"; also accept the legacy "NumberOfPagesToClose" @@ -311,6 +317,9 @@ func parseClosePageAction(raw map[string]any) *microflows.ClosePageAction { func parseShowMessageAction(raw map[string]any) *microflows.ShowMessageAction { action := µflows.ShowMessageAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.Blocking = extractBool(raw["Blocking"], false) if msgType, ok := raw["Type"].(string); ok { @@ -342,6 +351,9 @@ func parseShowMessageAction(raw map[string]any) *microflows.ShowMessageAction { func parseValidationFeedbackAction(raw map[string]any) *microflows.ValidationFeedbackAction { action := µflows.ValidationFeedbackAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.ObjectVariable = extractString(raw["ValidationVariableName"]) action.AttributeName = extractString(raw["Attribute"]) // BY_NAME_REFERENCE action.AssociationName = extractString(raw["Association"]) // BY_NAME_REFERENCE @@ -372,6 +384,9 @@ func parseDownloadFileAction(raw map[string]any) *microflows.DownloadFileAction func parseLogMessageAction(raw map[string]any) *microflows.LogMessageAction { action := µflows.LogMessageAction{} action.ID = model.ID(extractBsonID(raw["$ID"])) + // ErrorHandlingType gates the whole error branch in DESCRIBE, not just a + // suffix — see getActionErrorHandlingType (mendixlabs/mxcli#1078). + action.ErrorHandlingType = microflows.ErrorHandlingType(extractString(raw["ErrorHandlingType"])) action.LogNodeName = extractString(raw["Node"]) action.IncludeLastStackTrace = extractBool(raw["IncludeLatestStackTrace"], false) diff --git a/sdk/mpr/parser_microflow_error_handling_1078_test.go b/sdk/mpr/parser_microflow_error_handling_1078_test.go new file mode 100644 index 0000000000..431e5eb0d9 --- /dev/null +++ b/sdk/mpr/parser_microflow_error_handling_1078_test.go @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "testing" + + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1078, the legacy engine's half. +// +// Fixing the describer made the default (modelsdk) engine round-trip an error +// handler again, and MXCLI_ENGINE=legacy still dropped it — for a second, +// independent reason: nine parse functions never read ErrorHandlingType off the +// BSON at all, so the value was gone before the describer could be asked about +// it. Measured on the same project: 0 errors on modelsdk, handler still missing +// on legacy, until these were fixed too. +// +// The two defects are stacked, which is why fixing one looked like fixing both. +func TestParse1078_ActionsReadErrorHandlingType(t *testing.T) { + // "Custom" is what Studio Pro's "custom with rollback" stores, and it is the + // value the reporter's create-variable activity carried. + const custom = "Custom" + + for _, tc := range []struct { + name string + got func() microflows.ErrorHandlingType + }{ + {"create variable", func() microflows.ErrorHandlingType { + return parseCreateVariableAction(map[string]any{ + "$ID": "a", "VariableName": "name", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"change variable", func() microflows.ErrorHandlingType { + return parseChangeVariableAction(map[string]any{ + "$ID": "a", "ChangeVariableName": "name", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"create object", func() microflows.ErrorHandlingType { + return parseCreateObjectAction(map[string]any{ + "$ID": "a", "Entity": "Mod.Car", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"change object", func() microflows.ErrorHandlingType { + return parseChangeObjectAction(map[string]any{ + "$ID": "a", "ChangeVariableName": "Car", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"close page", func() microflows.ErrorHandlingType { + return parseClosePageAction(map[string]any{ + "$ID": "a", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"log message", func() microflows.ErrorHandlingType { + return parseLogMessageAction(map[string]any{ + "$ID": "a", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"show message", func() microflows.ErrorHandlingType { + return parseShowMessageAction(map[string]any{ + "$ID": "a", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"show page", func() microflows.ErrorHandlingType { + return parseShowPageAction(map[string]any{ + "$ID": "a", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + {"validation feedback", func() microflows.ErrorHandlingType { + return parseValidationFeedbackAction(map[string]any{ + "$ID": "a", "ErrorHandlingType": custom, + }).ErrorHandlingType + }}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := tc.got(); got != microflows.ErrorHandlingTypeCustom { + t.Errorf("ErrorHandlingType = %q, want %q — the whole error branch is "+ + "dropped from DESCRIBE when this is empty", + got, microflows.ErrorHandlingTypeCustom) + } + }) + } +} + +// Control. An absent ErrorHandlingType must stay empty rather than being invented: +// #840 established that a rendered `on error rollback` puts a clause in the +// user's script they never wrote, and these parsers are read by the same +// describer. +func TestParse1078_AbsentErrorHandlingTypeStaysEmpty(t *testing.T) { + if got := parseCreateVariableAction(map[string]any{ + "$ID": "a", "VariableName": "name", + }).ErrorHandlingType; got != "" { + t.Errorf("ErrorHandlingType = %q, want empty for BSON that carries none", got) + } +} From ba1bf6bac8b0db548e94ed31eb3b91e5b2d0c5ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 20:48:17 +0000 Subject: [PATCH 12/19] fix(check): report a duplicate CREATE MODULE ROLE against the project MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last gap of the class dbe5cc2a closed for documents. `create module role M.Admin` on a role that already exists passed `check --references` and then failed at exec with "module role already exists" — after every statement before it had been written, since mxcli does not run a script in one transaction. A module role is not a *document*, so `CreateModuleRoleStmt` was never in `stmtCreateInfo` and the sweep that compared stmtCreateInfo against `projectNameSets.setFor` never saw it. Worth naming precisely, because the guard that class produced is a good one and it stayed green through this: `TestEveryCreateDocTypeIsProjectChecked` reads both lists out of the Go source and compares them, and a type missing from BOTH lists is invisible to it. The complement it cannot express — every statement exec refuses with "already exists" is classified at all — has no list to read from. This one was found by running the statements, not by reading the code. Matching the exec predicate mattered more than the wiring. Two behaviours had to be reproduced or a fixed under-report would become a false positive: - CREATE OR MODIFY succeeds on an existing role — the `idempotent` flag. - A plain CREATE succeeds on a role mxcli AUTO-PROVISIONED. `execCreateModuleRole` adopts the caller's casing, rewrites the references and returns nil. So `buildModuleRoleQualifiedNames` excludes any role carrying `autoDocumentRoleDescription`; without that, check would refuse a working script over a role the user never asked mxcli to create (`defaultDocumentAccessRoles` adds it on the first document in a role-less module). Verified live with its control: the auto-provisioned role passes check and exec prints "already exists (auto-provisioned)", while an authored role in the same module is reported. `stmtDropInfo` gets the DROP case in the same pass — without it, `drop module role X; create module role X;` reads as a conflict. Classifying the type also gives the script-duplicate half for free, since both consumers share the name registry. Known limit, recorded rather than closed: exec matches role names case-insensitively (Mendix does, CE0123) and this set is keyed exactly, like every other doc type, so `create module role M.admin` against a stored `M.Admin` is still reported only by exec. That is an under-report of a rare spelling, not a false positive, and closing it would mean case-folding the shared registry for one type. Repro: mdl-examples/bug-tests/1067-duplicate-module-role.mdl — a plain .mdl, not a .fail.mdl: the check needs a project and `make check-mdl` runs check without one, so the negative-test naming would report it as regressed. Issue: mendixlabs/mxcli#1067 (follow-up) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017s476QkXr9CFMvKspVzcvu --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../bug-tests/1067-duplicate-module-role.mdl | 40 +++++++ mdl/executor/helpers.go | 36 ++++++ mdl/executor/validate_duplicates.go | 15 +++ .../validate_duplicates_module_role_test.go | 107 ++++++++++++++++++ 5 files changed, 199 insertions(+) create mode 100644 mdl-examples/bug-tests/1067-duplicate-module-role.mdl create mode 100644 mdl/executor/validate_duplicates_module_role_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 603291ee00..e5ec9d8e20 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -572,3 +572,4 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A widget-describe change measured as 'zero rules lost' had in fact dropped SIX visibility rules from Combo box — the widget the work was justified by. The before/after sweep enumerated widgets from `.mxcli/widgets/*.def.json` (33 of them) but `mxcli widget describe` also serves widgets straight from their `.mpk`, and Combo box has no def.json. The real describable set is 42; the missing 9 included the one that mattered.", "cause": "The sweep's widget list was derived from an artifact of the pipeline (generated def.json files) rather than from the capability under test (what `widget describe` can describe). Nothing compared the two lists, so the sample silently excluded a whole class — the same 'two lists, nothing comparing them' shape as the defects being fixed.", "file": "mdl/executor/editorconfig_extract.go, mdl/executor/editorconfig_shapes_test.go", "insight": "Derive a sweep's population from the CAPABILITY, never from a convenient artifact, and state the population in the claim: 'zero rules lost across the 33 widgets carrying a def.json' would have been true and would have invited the question. A second trap immediately after: the regression test written to lock the fix used a hand-written editorConfig snippet of the same APPARENT shape, and it passed with the fix reverted — the nesting that triggers the drop is three levels deep and specific, so the synthetic case was never flagged conjunctive and the test proved nothing. Pointing the test at the real committed .mpk made the control fail with all six names. Rule of thumb: when a defect was found in real vendor input, the regression test takes the real input; a reconstructed minimal case must be shown to fail without the fix BEFORE it is trusted, and here it did not. The fix itself is the policy that should have been there from the start: conjunction support may WITHHOLD a rule the extractor never produced before (emitting one conjunct over-fires), but may never drop one the older vocabulary already lifted — that rule's accuracy is unchanged by the new work.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "Six of Combo box's editorConfig hide-rules could not be lifted with their full condition, and its coverage counter sat at 21 of 32. The rules sit inside `\"association\"===t.optionsSourceType && ( … )`, itself the ELSE branch of a ternary inside `\"context\"===t.source ? ( … )` — a chained ternary, where each branch's BODY is parenthesised but each branch's CONDITION is not.", "cause": "groupGuard read the condition of a group opened after `&&`/`||` with trailingExpr, which stops at a STATEMENT separator. A chained ternary contains none, so it returned the whole `A ? (…) : B` expression as the group's condition. That is not a comparison, guardToCondition refused it, and enclosingGroupConditions reported the chain unreadable.", "file": "mdl/executor/editorconfig_extract.go (groupGuard, operandBefore)", "insight": "The characterisation written into the PR body — 'ternary chains without parentheses, which the outward walk does not traverse' — was WRONG, and instrumenting the walk rather than re-reading it is what showed so: the walk reaches these groups fine; the failure is guard EXTRACTION at the group, one function away. A one-line ceiling written from reasoning is worth re-deriving before anyone builds on it. The fix is not a straight swap to lastGuardExpr, which bounds at `{` and hands back a fragment with an unbalanced `}` where the expression follows a block (ProgressCircle's ternary follows a whole switch); take lastGuardExpr's answer only when it stopped at an INSIDE-expression boundary (`:`, `?`, `,`) and fall back to trailingExpr otherwise. Also a measurement note: the 'before' number quoted from an earlier session (16 of 32) was stale — the branch had been restarted from a main that already carried the previous fix, so the real baseline was 21. Re-measure the baseline in the tree you are actually editing rather than quoting a figure from memory.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "`alter page P { set NoSuchProperty = 10 on dgProducts; }` and `set PageSize = 12 on noSuchWidget;` both passed `mxcli check -p app.mpr --references` (exit 0, \"All references valid\") and were then refused by `exec`: `pluggable property \"NoSuchProperty\" not found` / `widget \"noSuchWidget\" not found`. exec applies statements one at a time, so the script had already written everything before the typo", "cause": "ValidateWidgetProperties resolves the properties of widgets a statement CARRIES \u2014 CREATE PAGE's tree, and ALTER's InsertWidgetOp/ReplaceWidgetOp trees. SetPropertyOp carries no widget: it names one already stored, so its property can only be resolved against the DOCUMENT, which that pass never opens. Same inversion validate_alter_target.go closed for the ALTER's target document, one level further in", "file": "`mdl/executor/validate_alter_set.go` (new), `mdl/backend/pagemutator/probe.go` (new)", "insight": "**Do not re-derive what a mutation accepts \u2014 run the mutation.** The vocabulary of an ALTER SET is partly a switch in `setRawWidgetPropertyMut` and partly the STORED widget's own PropertyTypes, which belong to whatever widget package the project installed; no registry in this repo can state it for an arbitrary project. So the check opens the document, runs the real setter against a throwaway deep copy (`Mutator.Probe`, whose `Save` is refused), and keeps only the error \u2014 check and exec cannot drift because there is one resolver. The author also gets exec's exact wording from the pre-flight. **Two false-positive sources, both measured, both silence rather than a finding**: a page the script CREATEs (nothing stored \u2014 skip, and do not even open it) and a widget an INSERT in the same script adds. The second cannot be a name match: a DataGrid 2 column is inserted as `colBrand` and addressed as `Brand` (derived from the bound attribute), so the rule is `ResolvesTarget` \u2014 suppress only when the document does not carry the target AND the script adds widgets to that document. **Gating on an optional interface assertion, not on backend.PageMutator**, keeps the pass off the MCP mutator, which has no pluggable path and would have reported its own difference as the author's mistake. Controls: 4 casings of a real property pass (a case-sensitive pre-flight would have re-broken #1069); insert-then-set passes check AND exec in both the same-statement and across-statement forms; 17 ALTER SET examples in mdl-examples show 0 new errors vs the baseline binary; the project's files are byte-identical (md5 over .mpr + mprcontents) after five check runs. Revert control: stubbing the pass makes the three gap tests fail", "refs": []} +{"area": "mdl/executor", "date": "2026-09-09", "symptom": "`mxcli check … --references` reports \"Check passed!\" for a script whose `create module role M.Admin` names a role that already exists; `exec` then fails with \"module role already exists: M.Admin\" — after the statements before it have been written, leaving the project half-modified.", "cause": "The last gap of the class dbe5cc2a closed for documents. A module role is not a *document*, so `CreateModuleRoleStmt` was never in `stmtCreateInfo` and the sweep that compared stmtCreateInfo against `projectNameSets.setFor` never saw it — the guard tests pass because both lists agree on the types they contain, and a type absent from BOTH is invisible to a test that only compares them.", "file": "`mdl/executor/validate_duplicates.go` (`stmtCreateInfo`, `stmtDropInfo`, `setFor`, `friendlyDocType`), `mdl/executor/helpers.go` (`buildModuleRoleQualifiedNames`)", "insight": "**A guard that compares two lists cannot catch a type missing from both.** `TestEveryCreateDocTypeIsProjectChecked` reads stmtCreateInfo and setFor out of the Go source and is exactly right about the defect it was written for, and it stayed green through this one. The complement it cannot express — every statement `exec` refuses with \"already exists\" is classified at all — has no list to read; it was found by running the statements, not by reading code. **Match the exec predicate exactly, or a fixed under-report becomes a false positive.** Two exec behaviours had to be reproduced: CREATE OR MODIFY succeeds (the `idempotent` flag), and a plain CREATE on a role mxcli AUTO-PROVISIONED also succeeds — `execCreateModuleRole` adopts the caller's casing, rewrites references and returns nil — so `buildModuleRoleQualifiedNames` excludes any role carrying `autoDocumentRoleDescription`. Without that, `check` refuses a working script over a role the user never asked mxcli to create (`defaultDocumentAccessRoles` adds it on the first document in a role-less module). Verified live with its control: auto-provisioned role → check clean and exec prints \"already exists (auto-provisioned)\"; an authored role in the SAME module → reported. **`stmtDropInfo` is the other half**: without the DROP case, `drop module role X; create module role X;` reads as a conflict. Known limit, recorded rather than closed: exec matches role names case-insensitively (CE0123) and this set is keyed exactly, like every other doc type, so `create module role M.admin` against a stored `M.Admin` is still reported only by exec — an under-report of a rare spelling, not a false positive. Repro `mdl-examples/bug-tests/1067-duplicate-module-role.mdl` (a plain .mdl: the check needs a project, and `make check-mdl` runs check without one). Issue mendixlabs/mxcli#1067", "refs": ["mendixlabs/mxcli#1067"]} diff --git a/mdl-examples/bug-tests/1067-duplicate-module-role.mdl b/mdl-examples/bug-tests/1067-duplicate-module-role.mdl new file mode 100644 index 0000000000..23e2850abb --- /dev/null +++ b/mdl-examples/bug-tests/1067-duplicate-module-role.mdl @@ -0,0 +1,40 @@ +-- mendixlabs/mxcli#1067, follow-up: a duplicate CREATE MODULE ROLE. +-- +-- The last survivor of the under-reporting class dbe5cc2a closed for documents. +-- `create module role M.Admin` on a role that already exists passed +-- `check --references` and then failed at exec — after every statement before it +-- had been written, since mxcli does not run a script in one transaction. A +-- module role is not a *document*, so it was never in stmtCreateInfo and fell +-- outside that sweep. +-- +-- Run this twice against the same project: +-- +-- mxcli exec 1067-duplicate-module-role.mdl -p app.mpr # first run: creates +-- mxcli check 1067-duplicate-module-role.mdl -p app.mpr --references +-- +-- The second command now reports: +-- statement 2: module role already exists in project: Zz1067d.Admin +-- — use CREATE OR MODIFY to update it +-- +-- This repro is a plain .mdl, not a .fail.mdl: the check needs a project to +-- decide anything, and `make check-mdl` runs `check` WITHOUT one. Naming it +-- .fail.mdl would report "negative test unexpectedly passed" — a working rule +-- made to look regressed. The guard itself is covered by unit tests in +-- mdl/executor/validate_duplicates_module_role_test.go. +-- +-- Two cases that must stay clean, and are the reason this is not a blanket +-- "role exists" check: +-- +-- * CREATE OR MODIFY (below) — the re-runnable spelling the docs recommend. +-- * A role mxcli AUTO-PROVISIONED. Creating any document in a module with no +-- roles makes mxcli add `User` itself; `create module role M.User` then +-- SUCCEEDS at exec ("already exists (auto-provisioned)"), adopting the +-- caller's casing. Flagging that would refuse a working script over a role +-- the user never asked for. + +create module Zz1067d; + +create module role Zz1067d.Admin description 'Full access'; + +-- Re-runnable: this one is never reported, on any run. +create or modify module role Zz1067d.Viewer description 'Read-only'; diff --git a/mdl/executor/helpers.go b/mdl/executor/helpers.go index f3f0ab65f2..d0945f7614 100644 --- a/mdl/executor/helpers.go +++ b/mdl/executor/helpers.go @@ -612,6 +612,42 @@ func buildAssociationQualifiedNames(ctx *ExecContext) map[string]bool { return result } +// buildModuleRoleQualifiedNames returns the module roles a plain +// CREATE MODULE ROLE would collide with, as Module.Role. +// +// Roles mxcli auto-provisioned are deliberately EXCLUDED. `execCreateModuleRole` +// treats one of those as a hit rather than a conflict — it adopts the caller's +// casing, rewrites the references, and returns nil — so listing it here would +// make `check` refuse a script that runs fine, and refuse it over a role the +// user never asked mxcli to create (`defaultDocumentAccessRoles`). A check that +// reports a statement exec accepts is worse than the under-report it replaces. +// +// Known limit: exec matches role names case-insensitively (Mendix does, CE0123), +// while this set is keyed exactly, like every other doc type's. So +// `create module role M.admin` against a stored `M.Admin` is still reported only +// by exec. That is an under-report of a rare spelling, not a false positive, and +// closing it would mean case-folding the shared name registry for one type. +func buildModuleRoleQualifiedNames(ctx *ExecContext) map[string]bool { + result := make(map[string]bool) + modules, err := getModulesFromCache(ctx) + if err != nil { + return result + } + for _, m := range modules { + ms, err := ctx.Backend.GetModuleSecurity(m.ID) + if err != nil || ms == nil { + continue + } + for _, mr := range ms.ModuleRoles { + if mr == nil || mr.Description == autoDocumentRoleDescription { + continue + } + result[m.Name+"."+mr.Name] = true + } + } + return result +} + // buildRuleQualifiedNames returns a set of all rule qualified names in the project. func buildRuleQualifiedNames(ctx *ExecContext) map[string]bool { result := make(map[string]bool) diff --git a/mdl/executor/validate_duplicates.go b/mdl/executor/validate_duplicates.go index 7b00e900c6..05c09ccac7 100644 --- a/mdl/executor/validate_duplicates.go +++ b/mdl/executor/validate_duplicates.go @@ -94,6 +94,11 @@ func stmtCreateInfo(stmt ast.Statement) (docType, name string, idempotent bool) switch s := stmt.(type) { case *ast.CreateModuleStmt: return "module", s.Name, false + case *ast.CreateModuleRoleStmt: + // Not a document, which is why it was missed when the document types + // were swept (mendixlabs/mxcli#1067). exec refuses a plain CREATE of an + // existing role, so check has to as well. + return "module-role", s.Name.String(), s.CreateOrModify case *ast.CreateEntityStmt: return "entity", s.Name.String(), s.CreateOrModify || s.IfNotExists case *ast.CreateViewEntityStmt: @@ -154,6 +159,8 @@ func stmtDropInfo(stmt ast.Statement) (docType, name string) { switch s := stmt.(type) { case *ast.DropModuleStmt: return "module", s.Name + case *ast.DropModuleRoleStmt: + return "module-role", s.Name.String() case *ast.DropEntityStmt: return "entity", s.Name.String() case *ast.DropEnumerationStmt: @@ -251,6 +258,8 @@ func friendlyDocType(docType string) string { return "java action" case "javascriptaction": return "javascript action" + case "module-role": + return "module role" case "json-structure": return "JSON structure" case "knowledge-base": @@ -359,6 +368,7 @@ type projectNameSets struct { associations map[string]bool rules map[string]bool javaScriptActs map[string]bool + moduleRoles map[string]bool } // projectSetFor returns the existence set for the given doc-type key, or nil @@ -411,6 +421,8 @@ func (ps *projectNameSets) setFor(docType string) map[string]bool { return ps.rules case "javascriptaction": return ps.javaScriptActs + case "module-role": + return ps.moduleRoles } // "module" is deliberately absent: CREATE MODULE on an existing module is a // no-op that prints "already exists" and exits 0, so `create module M;` is @@ -549,6 +561,9 @@ func loadProjectNameSets(ctx *ExecContext) *projectNameSets { // JavaScript actions ps.javaScriptActs = buildJavaScriptActionQualifiedNames(ctx) + // Module roles + ps.moduleRoles = buildModuleRoleQualifiedNames(ctx) + // Image collections ps.imageCollections = make(map[string]bool) if ics, err := ctx.Backend.ListImageCollections(); err == nil { diff --git a/mdl/executor/validate_duplicates_module_role_test.go b/mdl/executor/validate_duplicates_module_role_test.go new file mode 100644 index 0000000000..21d60d76bf --- /dev/null +++ b/mdl/executor/validate_duplicates_module_role_test.go @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The last survivor of the under-reporting class dbe5cc2a closed for documents: +// `create module role M.Admin` on a role that already exists passed +// `check --references` and then failed at exec, leaving every statement before +// it applied. A module role is not a *document*, so it was never in +// stmtCreateInfo and fell outside that sweep. +// +// Two exec behaviours the check has to match exactly, or it trades a silent +// under-report for a noisy false positive: +// +// - CREATE OR MODIFY succeeds on an existing role. +// - A plain CREATE succeeds on a role mxcli AUTO-PROVISIONED (`User`, carrying +// autoDocumentRoleDescription). execCreateModuleRole adopts the caller's +// casing and returns nil, so a check that flagged it would be wrong. +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// setupModuleRoleConflictCtx gives module M two roles: "Admin", authored, and +// "User", auto-provisioned by mxcli. +func setupModuleRoleConflictCtx(t *testing.T) *ExecContext { + t.Helper() + mod := mkModule("M") + h := mkHierarchy(mod) + + ms := &security.ModuleSecurity{ + BaseElement: model.BaseElement{ID: nextID("ms")}, + ContainerID: mod.ID, + ModuleRoles: []*security.ModuleRole{ + {BaseElement: model.BaseElement{ID: nextID("mr")}, Name: "Admin", Description: "Authored by a person"}, + {BaseElement: model.BaseElement{ID: nextID("mr")}, Name: "User", Description: autoDocumentRoleDescription}, + }, + } + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(model.ID) (*security.ModuleSecurity, error) { + return ms, nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + return ctx +} + +// The defect itself: exec refuses this, so check must too. +func TestProjectConflicts_ModuleRoleAlreadyExists(t *testing.T) { + ctx := setupModuleRoleConflictCtx(t) + assertHasConflict(t, ctx, `create module role M.Admin;`, "M.Admin") +} + +// A role that is not there is not a conflict. +func TestProjectConflicts_NewModuleRoleIsClean(t *testing.T) { + ctx := setupModuleRoleConflictCtx(t) + assertNoConflicts(t, ctx, `create module role M.Auditor;`) +} + +// The re-runnable spelling. Flagging it would break every security script that +// is written to be replayed, which is the form the docs recommend. +func TestProjectConflicts_CreateOrModifyModuleRoleIsClean(t *testing.T) { + ctx := setupModuleRoleConflictCtx(t) + assertNoConflicts(t, ctx, `create or modify module role M.Admin description 'x';`) +} + +// The false positive worth guarding: exec SUCCEEDS on an auto-provisioned role, +// adopting the caller's casing. A check that reported it would refuse a script +// that works — and mxcli created that role itself, without being asked. +func TestProjectConflicts_AutoProvisionedModuleRoleIsClean(t *testing.T) { + ctx := setupModuleRoleConflictCtx(t) + assertNoConflicts(t, ctx, `create module role M.User;`) +} + +// A script that drops the role first is re-creating it, not colliding with it. +// Without DROP MODULE ROLE in stmtDropInfo this reads as a conflict. +func TestProjectConflicts_DropThenCreateModuleRoleIsClean(t *testing.T) { + ctx := setupModuleRoleConflictCtx(t) + assertNoConflicts(t, ctx, ` + drop module role M.Admin; + create module role M.Admin; + `) +} + +// Two CREATEs of the same new role in one script collide with each other even +// though neither exists in the project. This is the OTHER consumer of the same +// registry — CheckScriptDuplicates, which needs no project — and it comes for +// free once the type is classified. +func TestScriptDuplicates_ModuleRoleTwiceInOneScript(t *testing.T) { + assertHasDupViolation(t, ` + create module role M.Auditor; + create module role M.Auditor; + `, "M.Auditor") +} + +// …and the re-runnable spelling must stay clean there too. +func TestScriptDuplicates_CreateOrModifyModuleRoleTwiceIsClean(t *testing.T) { + assertNoDupViolations(t, ` + create or modify module role M.Auditor; + create or modify module role M.Auditor; + `) +} From c99eab69929e171c3bde2b15b30ed8361b589ca6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 06:13:05 +0000 Subject: [PATCH 13/19] test(gate): run the legacy engine nightly, not on every push MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doctype gate hands every script to mxbuild once PER ENGINE, and mxbuild dominates the cost, so the second engine is most of what the gate spends. Measured back to back, both -count=1: the doctype gate takes 588.5s over the full matrix and 232.2s over modelsdk alone — 61% off, for the engine that is no longer the default. MXCLI_TEST_ENGINES narrows the matrix. The per-push CI job sets modelsdk; the nightly sets all, and already runs across a Mendix-version matrix, so legacy is verified daily rather than never. Not deleted, and not disabled. The legacy engine still ships and is still selectable (--engine legacy), and five fallthroughs in widget_write.go still tell users to reach for it. Deleting its coverage while it ships would leave unverified code behind a documented path — the failure this repo has shipped once already (#808, an integration test that had only ever skipped). It is also the parity reference: establishing the SOAP writer meant diffing modelsdk's document against legacy's, because no Studio Pro-authored SOAP document exists here. Coverage should go last, when there is nothing left to cover. Two decisions worth keeping: - The DEFAULT is every engine. Nightly could have relied on a default of "modelsdk" and asked for "all" itself, but then a mistake in EITHER workflow file loses legacy coverage silently. With this default a mistake in the per-push file costs minutes, not coverage. - An unrecognised name is fatal, not ignored. A typo that selected nothing would run zero scripts and report the gate green, which is the one outcome a gate must never have. TestMain refuses to start, and a narrowed matrix announces itself in the log so a run never implies coverage it did not have. Verified: narrowed run executes 68 subtests and 0 legacy ones, prints the narrowing notice, exits 0; the full matrix executes 136 and prints no notice. (The first control I ran was invalid — Go served a cached result — so both figures above are from -count=1 runs.) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .github/workflows/nightly.yml | 6 ++ .github/workflows/push-test.yml | 8 ++ Makefile | 7 ++ mdl/executor/roundtrip_doctype_test.go | 66 +++++++++++++- mdl/executor/roundtrip_engine_matrix_test.go | 91 ++++++++++++++++++++ mdl/executor/roundtrip_helpers_test.go | 14 +++ 6 files changed, 191 insertions(+), 1 deletion(-) create mode 100644 mdl/executor/roundtrip_engine_matrix_test.go diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 587a737238..4fb2ea6cde 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -44,8 +44,14 @@ jobs: - name: Setup mxbuild ${{ matrix.mendix-version }} run: ./bin/mxcli setup mxbuild --version ${{ matrix.mendix-version }} + # The full engine matrix runs here, not on every push. "all" is the + # default, so this is belt and braces — but it is the line that makes the + # nightly the place legacy is verified, and it should survive someone + # changing the default. - name: Integration tests (Mendix ${{ matrix.mendix-version }}) run: make test-integration + env: + MXCLI_TEST_ENGINES: all timeout-minutes: 30 nightly: diff --git a/.github/workflows/push-test.yml b/.github/workflows/push-test.yml index f23df9d01d..6964c72463 100644 --- a/.github/workflows/push-test.yml +++ b/.github/workflows/push-test.yml @@ -102,8 +102,16 @@ jobs: run: ./scripts/check-skill-mdl.sh ./bin/mxcli docs-site/src - name: Setup mxbuild run: ./bin/mxcli setup mxbuild --version 11.12.2 + # Per-push, the gate runs the DEFAULT engine only. The doctype gate hands + # every script to mxbuild once per engine and mxbuild dominates the cost, + # so the second engine is roughly half of this step. The legacy engine is + # still shipped and still selectable (--engine legacy), so it is not + # untested: the nightly runs the full matrix, across every Mendix version. + # Drop this env to bring legacy back to every push. - name: Integration tests run: make test-integration + env: + MXCLI_TEST_ENGINES: modelsdk timeout-minutes: 30 - name: Lint Go run: make lint-go diff --git a/Makefile b/Makefile index 540a079925..6c13eee20f 100644 --- a/Makefile +++ b/Makefile @@ -286,6 +286,13 @@ check-tunnel-deps: @./scripts/check-tunnel-deps.sh # Run integration tests (requires mx binary / mxbuild) +# +# The gate runs every doctype script through exec + mx check once PER ENGINE. +# MXCLI_TEST_ENGINES narrows that matrix — `MXCLI_TEST_ENGINES=modelsdk make +# test-integration` skips the legacy engine and takes roughly 60% off the +# doctype gate (measured: 588s -> 232s). Unset means every engine, which is what +# the nightly runs; the per-push CI job narrows it to modelsdk. An unrecognised +# engine name is fatal rather than silently selecting nothing. test-integration: CGO_ENABLED=0 go test -tags integration -count=1 -timeout 30m ./... diff --git a/mdl/executor/roundtrip_doctype_test.go b/mdl/executor/roundtrip_doctype_test.go index 4a61bd1849..a3fee68379 100644 --- a/mdl/executor/roundtrip_doctype_test.go +++ b/mdl/executor/roundtrip_doctype_test.go @@ -33,11 +33,75 @@ type gateEngine struct { factory func() backend.FullBackend } -var gateEngines = []gateEngine{ +var allGateEngines = []gateEngine{ {"modelsdk", func() backend.FullBackend { return modelsdkbackend.New() }}, {"legacy", func() backend.FullBackend { return mprbackend.New() }}, } +// gateEnginesEnv narrows the matrix above to a subset, as a comma- or +// space-separated list of engine names ("modelsdk", "legacy"); empty or "all" +// means every engine. +// +// It exists because the matrix is most of what the gate costs: each script is +// executed and then handed to mxbuild once PER ENGINE, and mxbuild dominates. +// On CI the per-push job runs `modelsdk` alone (see .github/workflows/ +// push-test.yml) and the nightly runs the full matrix across the Mendix-version +// matrix (nightly.yml), so legacy stays verified daily without every push +// paying for it. +// +// The DEFAULT is every engine, deliberately. Nightly could have relied on a +// default of "modelsdk" and set "all" itself, but then a mistake in EITHER +// workflow file loses legacy coverage silently, and a lost gate is the failure +// this repo has already shipped once (#808, an integration test that had only +// ever skipped). With this default a mistake in the per-push file costs +// minutes, not coverage — the failure mode is biased the right way. For the +// same reason a narrowed matrix is announced in TestMain rather than applied +// quietly: a run that covered less than it looks like it did should say so. +const gateEnginesEnv = "MXCLI_TEST_ENGINES" + +// gateEngines is the matrix every gate test loops over. +var gateEngines, unknownGateEngines = selectGateEngines(os.Getenv(gateEnginesEnv), allGateEngines) + +// selectGateEngines filters the engine matrix by name, preserving matrix order +// so the subset runs in the same sequence as the whole. Unrecognised names are +// returned rather than ignored: a typo that silently selects NOTHING would turn +// the gate into a no-op that still reports success, which is the one outcome a +// gate must never have. +func selectGateEngines(spec string, all []gateEngine) (selected []gateEngine, unknown []string) { + fields := strings.FieldsFunc(spec, func(r rune) bool { return r == ',' || r == ' ' || r == '\t' }) + if len(fields) == 0 { + return all, nil + } + wanted := make(map[string]bool, len(fields)) + for _, f := range fields { + name := strings.ToLower(strings.TrimSpace(f)) + if name == "all" { + return all, nil + } + wanted[name] = true + } + for _, eng := range all { + if wanted[eng.name] { + selected = append(selected, eng) + delete(wanted, eng.name) + } + } + for name := range wanted { + unknown = append(unknown, name) + } + sort.Strings(unknown) + return selected, unknown +} + +// gateEngineNames renders an engine list for a log line. +func gateEngineNames(engines []gateEngine) string { + names := make([]string, 0, len(engines)) + for _, e := range engines { + names = append(names, e.name) + } + return strings.Join(names, ", ") +} + // engineScriptSkip marks (engine/script) pairs to skip, with a reason. Use only // for a script that fails on ONE engine for a tracked, not-yet-actionable reason // (e.g. a known modelsdk gap on a specific document type). A script broken on diff --git a/mdl/executor/roundtrip_engine_matrix_test.go b/mdl/executor/roundtrip_engine_matrix_test.go new file mode 100644 index 0000000000..da0ded1ae4 --- /dev/null +++ b/mdl/executor/roundtrip_engine_matrix_test.go @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// The engine matrix decides how much the gate actually covers, so its selection +// is worth testing directly — the alternative is discovering in a CI log that a +// run "passed" over nothing. + +func testMatrix() []gateEngine { + return []gateEngine{ + {"modelsdk", func() backend.FullBackend { return nil }}, + {"legacy", func() backend.FullBackend { return nil }}, + } +} + +func TestSelectGateEngines(t *testing.T) { + for _, tc := range []struct { + name string + spec string + want string + unknown string + }{ + // The default has to be the FULL matrix: every workflow that wants less + // says so, and a file that forgets loses minutes rather than coverage. + {"empty is every engine", "", "modelsdk, legacy", ""}, + {"whitespace is every engine", " ", "modelsdk, legacy", ""}, + {"all is every engine", "all", "modelsdk, legacy", ""}, + {"single engine", "modelsdk", "modelsdk", ""}, + {"the other engine", "legacy", "legacy", ""}, + {"comma separated", "modelsdk,legacy", "modelsdk, legacy", ""}, + {"space separated", "modelsdk legacy", "modelsdk, legacy", ""}, + // Order follows the matrix, not the spec, so a subset runs in the same + // sequence as the whole and logs line up between runs. + {"order follows the matrix", "legacy,modelsdk", "modelsdk, legacy", ""}, + {"case and padding are tolerated", " MODELSDK , legacy ", "modelsdk, legacy", ""}, + {"all wins over a narrower name", "legacy,all", "modelsdk, legacy", ""}, + // The one that matters: a typo must be reported, never silently + // selecting nothing. + {"a typo is reported", "modelsdkk", "", "modelsdkk"}, + {"a typo beside a real name is reported", "modelsdk,legcy", "modelsdk", "legcy"}, + } { + t.Run(tc.name, func(t *testing.T) { + got, unknown := selectGateEngines(tc.spec, testMatrix()) + if name := gateEngineNames(got); name != tc.want { + t.Errorf("selected = %q, want %q", name, tc.want) + } + if u := strings.Join(unknown, ", "); u != tc.unknown { + t.Errorf("unknown = %q, want %q", u, tc.unknown) + } + }) + } +} + +// TestGateEnginesIsNeverSilentlyEmpty is the property TestMain's fatal check +// rests on: a spec that selects no engine must be distinguishable from one that +// selects every engine. Without this, `MXCLI_TEST_ENGINES=modelsdkk` would run +// zero scripts and report the gate green. +func TestGateEnginesIsNeverSilentlyEmpty(t *testing.T) { + selected, unknown := selectGateEngines("nosuchengine", testMatrix()) + if len(selected) != 0 { + t.Fatalf("selected %d engines for an unknown name, want 0", len(selected)) + } + if len(unknown) == 0 { + t.Fatal("an unknown engine name produced no report — the gate would run nothing and pass") + } +} + +// TestGateEnginesMatchesTheProcessEnv confirms the wiring: the live matrix is +// the one selectGateEngines produced, not a separately maintained list. +func TestGateEnginesMatchesTheProcessEnv(t *testing.T) { + if len(unknownGateEngines) != 0 { + t.Fatalf("this run has unknown engine names %v — TestMain should have refused to start", unknownGateEngines) + } + if len(gateEngines) == 0 { + t.Fatal("the live engine matrix is empty") + } + for _, eng := range gateEngines { + if eng.factory == nil { + t.Errorf("engine %q has no backend factory", eng.name) + } + } +} diff --git a/mdl/executor/roundtrip_helpers_test.go b/mdl/executor/roundtrip_helpers_test.go index 85bc1cfccd..0b2d1d4f43 100644 --- a/mdl/executor/roundtrip_helpers_test.go +++ b/mdl/executor/roundtrip_helpers_test.go @@ -50,6 +50,20 @@ var sharedSourceMPR string // TestMain creates or locates the source project once, then runs all tests. // This avoids running `mx create-project` per test (~29s each). func TestMain(m *testing.M) { + // 0. Settle the engine matrix before anything runs. A narrowed matrix is + // announced rather than applied quietly, so a log never implies coverage the + // run did not have; an unrecognised name is fatal, because the alternative + // is a gate that selects no engine, runs nothing, and reports success. + if len(unknownGateEngines) > 0 { + fmt.Fprintf(os.Stderr, "FAIL: %s names unknown engine(s): %s (known: %s)\n", + gateEnginesEnv, strings.Join(unknownGateEngines, ", "), gateEngineNames(allGateEngines)) + os.Exit(1) + } + if len(gateEngines) != len(allGateEngines) { + fmt.Fprintf(os.Stderr, "TestMain: engine matrix narrowed by %s to %s (full matrix: %s)\n", + gateEnginesEnv, gateEngineNames(gateEngines), gateEngineNames(allGateEngines)) + } + // 1. Try the committed source project srcDir, err := filepath.Abs(sourceProject) if err == nil { From 185305def3da4ea5093ca9d8a0ce46419108ed7e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 06:32:21 +0000 Subject: [PATCH 14/19] fix(ci): carry the engine set in the step name, and correct a claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to the previous commit, both found by checking rather than by rereading. The step names now state the engine set, and the YAML is quoted. The first version was `- name: Integration tests (engines: modelsdk)`, which is INVALID YAML — an unquoted `: ` inside a scalar — and would have failed the whole workflow rather than the step. A yamllint pass caught it before it was pushed. The reason for the step name is the correction that matters. The previous commit claimed a narrowed matrix "announces itself in the log so a run never implies coverage it did not have". Measured: a full `make test-integration` with the matrix narrowed contains ZERO TestMain output, because `go test` without -v discards a passing package's stdout and stderr. The notice only ever reached my own -v verification runs, which is exactly why I believed it. The safety half was never affected: an unknown engine name exits non-zero, and a failing package's output IS shown, so that message lands. It was only the visibility half that was wrong, and the step name now carries it in the place it was claimed to work — a green CI log. The comments in roundtrip_doctype_test.go and roundtrip_helpers_test.go that stated the original claim are corrected to say which runs the notice reaches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .github/workflows/nightly.yml | 2 +- .github/workflows/push-test.yml | 2 +- mdl/executor/roundtrip_doctype_test.go | 7 +++++++ mdl/executor/roundtrip_helpers_test.go | 10 ++++++---- 4 files changed, 15 insertions(+), 6 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 4fb2ea6cde..ccf696e23c 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -48,7 +48,7 @@ jobs: # default, so this is belt and braces — but it is the line that makes the # nightly the place legacy is verified, and it should survive someone # changing the default. - - name: Integration tests (Mendix ${{ matrix.mendix-version }}) + - name: "Integration tests (Mendix ${{ matrix.mendix-version }}, engines: all)" run: make test-integration env: MXCLI_TEST_ENGINES: all diff --git a/.github/workflows/push-test.yml b/.github/workflows/push-test.yml index 6964c72463..5f83682dce 100644 --- a/.github/workflows/push-test.yml +++ b/.github/workflows/push-test.yml @@ -108,7 +108,7 @@ jobs: # still shipped and still selectable (--engine legacy), so it is not # untested: the nightly runs the full matrix, across every Mendix version. # Drop this env to bring legacy back to every push. - - name: Integration tests + - name: "Integration tests (engines: modelsdk)" run: make test-integration env: MXCLI_TEST_ENGINES: modelsdk diff --git a/mdl/executor/roundtrip_doctype_test.go b/mdl/executor/roundtrip_doctype_test.go index a3fee68379..d096e5897c 100644 --- a/mdl/executor/roundtrip_doctype_test.go +++ b/mdl/executor/roundtrip_doctype_test.go @@ -57,6 +57,13 @@ var allGateEngines = []gateEngine{ // minutes, not coverage — the failure mode is biased the right way. For the // same reason a narrowed matrix is announced in TestMain rather than applied // quietly: a run that covered less than it looks like it did should say so. +// +// One limit on that announcement, measured rather than assumed: `go test` +// without -v DISCARDS a passing package's output, so TestMain's notice does not +// reach a green CI log — only a -v run or a FAILING package shows it. The fatal +// path is unaffected (an unknown name exits non-zero, and a failing package's +// output is shown), but the visibility half is carried by the CI step NAME, +// which states the engine set outright. const gateEnginesEnv = "MXCLI_TEST_ENGINES" // gateEngines is the matrix every gate test loops over. diff --git a/mdl/executor/roundtrip_helpers_test.go b/mdl/executor/roundtrip_helpers_test.go index 0b2d1d4f43..627a09d681 100644 --- a/mdl/executor/roundtrip_helpers_test.go +++ b/mdl/executor/roundtrip_helpers_test.go @@ -50,10 +50,12 @@ var sharedSourceMPR string // TestMain creates or locates the source project once, then runs all tests. // This avoids running `mx create-project` per test (~29s each). func TestMain(m *testing.M) { - // 0. Settle the engine matrix before anything runs. A narrowed matrix is - // announced rather than applied quietly, so a log never implies coverage the - // run did not have; an unrecognised name is fatal, because the alternative - // is a gate that selects no engine, runs nothing, and reports success. + // 0. Settle the engine matrix before anything runs. An unrecognised name is + // fatal, because the alternative is a gate that selects no engine, runs + // nothing, and reports success — and a failing package's output IS shown, so + // this message lands. The narrowing notice below only reaches a -v run, since + // `go test` discards a passing package's output; the CI step name carries it + // for everyone else. if len(unknownGateEngines) > 0 { fmt.Fprintf(os.Stderr, "FAIL: %s names unknown engine(s): %s (known: %s)\n", gateEnginesEnv, strings.Join(unknownGateEngines, ", "), gateEngineNames(allGateEngines)) From 67100942842a663ea76fe33d0329af6f974f9934 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 06:32:37 +0000 Subject: [PATCH 15/19] fix(microflow): keep the nanoflow error-handling default (mendixlabs/mxcli#1078) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI caught a regression the previous commit introduced. `make test-integration` failed with 11 x CE6035 "Error handling type is not supported" across 02b-nanoflow-examples.mdl (both engines) and 03-page-examples.mdl, on every un-annotated Change object / Log message / Validation feedback / Close page activity in a nanoflow. `go test ./...` was green throughout — those mx-check round trips are behind `-tags integration`, which I had not run. Cause: eight builders were switched from `fb.ehType(nil)` to `explicitErrorHandling(fb, s.ErrorHandling)` on the action. explicitErrorHandling returns EMPTY for "no clause" and the writers turn empty into a literal "Rollback" — but fb.ehType is context-dependent and returns Abort in a nanoflow. So every un-annotated nanoflow activity silently changed from Abort to Rollback, which mxbuild rejects. The two helpers are not interchangeable, and which is right depends on what the call site did before. Retrieve and Delete legitimately use explicitErrorHandling: their writers emitted a hardcoded "Rollback" that those two actions accept in every flow flavour, so empty is a no-op there. Anywhere the builder already supplied a default, returning empty discards the flow flavour. That reasoning now lives on ehType's doc comment rather than being rediscovered. Reverted the eight sites to fb.ehType(s.ErrorHandling), which is byte-identical to the old behaviour when no clause is written and still carries an explicit one. Regression test TestAuthorOnError_NanoflowKeepsAbortWithoutAClause, with the same statements in a microflow as the control — a unit test, so it catches this in seconds rather than 30 minutes. The earlier control test asserted the empty string and so encoded the bug; it now asserts the flow's real default. Two adjacent gaps the same change opened, both measured on 11.14.0 rather than guessed: - A nanoflow accepts error handling on almost nothing. Only the two VARIABLE activities take a clause; change, log, show page, close page, show message and validation feedback are CE6035 whichever form is written, because the only accepted value there is Abort and no MDL syntax writes it. Refused now by checkNanoflowErrorHandling rather than written into a nanoflow mxbuild rejects. The split is by activity, not by client-side vs server-side: show message is as client-side as it gets and still refuses one. - getErrorHandling in nanoflow_validation.go gates the walk that looks for disallowed actions INSIDE a handler body, so the eight new statements needed adding there too — otherwise a Java action nested in `declare ... on error { ... }` goes unreported. Its test carries a commit control, which is how I found that the exported ValidateNanoflowBody is a different check from the unexported walk exec actually runs. Verified: go test ./... 84 ok / 0 fail; make lint clean; and this time `go test -tags integration ./...` — 84 ok, 0 fail, exit 0, including mdl/executor at 1209s, the package CI failed on. The #1078 round trip still holds end to end (both engines byte-identical, 0 errors over baseline on mxbuild 11.14.0). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 2 + .../skills/mendix/write-microflows/SKILL.md | 5 + .../skills/mendix/write-nanoflows/SKILL.md | 17 ++ cmd/mxcli/syntax/features_microflow.go | 5 + docs/01-project/MDL_QUICK_REFERENCE.md | 2 +- ...microflow-1078-error-handler-roundtrip.mdl | 8 + .../cmd_microflows_builder_actions.go | 15 +- mdl/executor/cmd_microflows_builder_calls.go | 25 ++- mdl/executor/cmd_microflows_builder_flows.go | 17 ++ .../microflow_error_handler_authoring_test.go | 187 +++++++++++++++++- mdl/executor/nanoflow_validation.go | 101 ++++++++++ 11 files changed, 360 insertions(+), 24 deletions(-) diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 9552e24336..60d328fc85 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -573,3 +573,5 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "Six of Combo box's editorConfig hide-rules could not be lifted with their full condition, and its coverage counter sat at 21 of 32. The rules sit inside `\"association\"===t.optionsSourceType && ( … )`, itself the ELSE branch of a ternary inside `\"context\"===t.source ? ( … )` — a chained ternary, where each branch's BODY is parenthesised but each branch's CONDITION is not.", "cause": "groupGuard read the condition of a group opened after `&&`/`||` with trailingExpr, which stops at a STATEMENT separator. A chained ternary contains none, so it returned the whole `A ? (…) : B` expression as the group's condition. That is not a comparison, guardToCondition refused it, and enclosingGroupConditions reported the chain unreadable.", "file": "mdl/executor/editorconfig_extract.go (groupGuard, operandBefore)", "insight": "The characterisation written into the PR body — 'ternary chains without parentheses, which the outward walk does not traverse' — was WRONG, and instrumenting the walk rather than re-reading it is what showed so: the walk reaches these groups fine; the failure is guard EXTRACTION at the group, one function away. A one-line ceiling written from reasoning is worth re-deriving before anyone builds on it. The fix is not a straight swap to lastGuardExpr, which bounds at `{` and hands back a fragment with an unbalanced `}` where the expression follows a block (ProgressCircle's ternary follows a whole switch); take lastGuardExpr's answer only when it stopped at an INSIDE-expression boundary (`:`, `?`, `,`) and fall back to trailingExpr otherwise. Also a measurement note: the 'before' number quoted from an earlier session (16 of 32) was stale — the branch had been restarted from a main that already carried the previous fix, so the real baseline was 21. Re-measure the baseline in the tree you are actually editing rather than quoting a figure from memory.", "refs": ["mendixlabs/mxcli#1036"]} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "`alter page P { set NoSuchProperty = 10 on dgProducts; }` and `set PageSize = 12 on noSuchWidget;` both passed `mxcli check -p app.mpr --references` (exit 0, \"All references valid\") and were then refused by `exec`: `pluggable property \"NoSuchProperty\" not found` / `widget \"noSuchWidget\" not found`. exec applies statements one at a time, so the script had already written everything before the typo", "cause": "ValidateWidgetProperties resolves the properties of widgets a statement CARRIES \u2014 CREATE PAGE's tree, and ALTER's InsertWidgetOp/ReplaceWidgetOp trees. SetPropertyOp carries no widget: it names one already stored, so its property can only be resolved against the DOCUMENT, which that pass never opens. Same inversion validate_alter_target.go closed for the ALTER's target document, one level further in", "file": "`mdl/executor/validate_alter_set.go` (new), `mdl/backend/pagemutator/probe.go` (new)", "insight": "**Do not re-derive what a mutation accepts \u2014 run the mutation.** The vocabulary of an ALTER SET is partly a switch in `setRawWidgetPropertyMut` and partly the STORED widget's own PropertyTypes, which belong to whatever widget package the project installed; no registry in this repo can state it for an arbitrary project. So the check opens the document, runs the real setter against a throwaway deep copy (`Mutator.Probe`, whose `Save` is refused), and keeps only the error \u2014 check and exec cannot drift because there is one resolver. The author also gets exec's exact wording from the pre-flight. **Two false-positive sources, both measured, both silence rather than a finding**: a page the script CREATEs (nothing stored \u2014 skip, and do not even open it) and a widget an INSERT in the same script adds. The second cannot be a name match: a DataGrid 2 column is inserted as `colBrand` and addressed as `Brand` (derived from the bound attribute), so the rule is `ResolvesTarget` \u2014 suppress only when the document does not carry the target AND the script adds widgets to that document. **Gating on an optional interface assertion, not on backend.PageMutator**, keeps the pass off the MCP mutator, which has no pluggable path and would have reported its own difference as the author's mistake. Controls: 4 casings of a real property pass (a case-sensitive pre-flight would have re-broken #1069); insert-then-set passes check AND exec in both the same-statement and across-statement forms; 17 ALTER SET examples in mdl-examples show 0 new errors vs the baseline binary; the project's files are byte-identical (md5 over .mpr + mprcontents) after five check runs. Revert control: stubbing the pass makes the three gap tests fail", "refs": []} {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A microflow whose activity has a CUSTOM error handler comes back from `describe microflow` with the handler AND every activity in its branch missing. Output stays valid MDL, `mxcli check` passes, nothing on stderr \u2014 so describe\u2192edit\u2192exec deletes the handler from the model. Reported on a create-variable activity with \"custom with rollback\" (v0.21, Mx 11.12.3); reproduced on HEAD/11.14.0 with a pure mxcli round trip", "cause": "TWO STACKED DEFECTS. (1) `getActionErrorHandlingType` was a hand-maintained switch covering 17 of the 38 action types that store ErrorHandlingType. `emitActivityStatement` walks the error branch only when `hasCustomErrorHandler(errType)` agrees, so each of the 21 missing types lost the ENTIRE `on error { \u2026 }` block, not just the suffix \u2014 CreateObject and ChangeObject among them. (2) legacy only: 9 parse functions never read ErrorHandlingType off the BSON, so the value was gone before the describer was asked. Fixing (1) alone left legacy still broken, which is how they hid each other", "file": "`mdl/executor/cmd_microflows_show_helpers.go` (`getActionErrorHandlingType` \u2192 new `actionErrorHandlingField`); `sdk/mpr/parser_microflow.go`, `sdk/mpr/parser_microflow_actions.go` (9 parsers); grammar+visitor+builder for 8 statements; `mdl/executor/validate_microflow_error_handling.go` (MDL076 table, new MDL077)", "insight": "**Count the gap before fixing the instance** \u2014 the reported activity was 1 of 21, measured by diffing the switch's cases against the action types declaring the field. **Replace the enumeration, do not extend it**: the list had already been patched per-instance (#863) and silently regrew, so it is now a reflection lookup on the `ErrorHandlingType` field, with `RestOperationCallAction` the ONE deliberate exclusion (Mendix rejects a custom handler there, CE6035). Actions embed model.BaseElement, which has no such field, so no promoted field is picked up by accident. **The trap: fixing the describer alone makes things WORSE.** 8 statement forms had no `onErrorClause` in the grammar \u2014 `declare` (the reporter's own), `set`, `change`, `log`, `show page`, `close page`, `show message`, `validation feedback` \u2014 so DESCRIBE began emitting `declare $name String = 'v' on error { \u2026 };`, which fails to parse (`mismatched input 'on' expecting ';'`). Trading a silent drop for a broken script is not a fix; the grammar was extended instead. **Measured on 11.14.0, and the result is not guessable**: all 8 accept a custom handler (0 errors), but for `on error continue` create-VARIABLE and change-VARIABLE are fine while change-OBJECT, log, show page, close page, show message and validation feedback are CE6035 \u2014 now in MDL076's deny-list, whose comment claiming Log/Change were \"unreachable from a script\" this change invalidated. New MDL077 refuses `on error` on the list-operation/aggregate forms of `set`, which genuinely have no ErrorHandlingType in the metamodel \u2014 one MDL keyword spanning activities that can and cannot hold the clause. **A non-terminating handler causing CE0108 is NOT a bug**: the branch merges back and a later variable is out of scope on the error path; Studio Pro reports the same. Controls: revert the lookup \u2192 all 7 describer cases fail naming the dropped branch; revert one parser \u2192 the legacy case fails; a no-clause microflow must still render NO suffix (#840 in reverse). Repro `mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl`; describe\u2192exec\u2192describe byte-identical and reported \"Unchanged microflow\"", "refs": ["#1078", "#863", "#840"], "ce": ["CE6035", "CE0108"]} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "CI `make test-integration` fails where `go test ./...` is green: `TestMxCheck_DoctypeScripts/02b-nanoflow-examples.mdl` (both engines) and `/03-page-examples.mdl` report **CE6035 \"Error handling type is not supported\"** on 11 activities \u2014 every un-annotated Change object / Log message / Validation feedback / Close page in a NANOFLOW. Self-inflicted while fixing #1078", "cause": "Eight builders were switched from `fb.ehType(nil)` to `explicitErrorHandling(fb, s.ErrorHandling)` on the action. `explicitErrorHandling` returns EMPTY for \"no clause\", and the writers turn empty into a literal `\"Rollback\"` via orDefault \u2014 but `fb.ehType(nil)` is CONTEXT-DEPENDENT and returns **Abort** in a nanoflow. So every un-annotated nanoflow activity silently changed from Abort to Rollback, which mxbuild rejects", "file": "`mdl/executor/cmd_microflows_builder_actions.go` + `cmd_microflows_builder_calls.go` (8 sites reverted to `fb.ehType(s.ErrorHandling)`); `mdl/executor/nanoflow_validation.go` (`getErrorHandling`)", "insight": "**The same helper is correct for one action and wrong for its neighbour, and the difference is what the code did BEFORE.** `explicitErrorHandling` is right for Retrieve/Delete (#1020-era): their writers emitted a hardcoded `\"Rollback\"` that those two actions accept in every flow flavour, so empty\u2192Rollback is a no-op. It is wrong wherever the builder already supplied a *context-dependent* default \u2014 empty discards the flow flavour. The rule: before replacing a default-supplying expression, ask what the OLD expression returned in every context, not just the one you are testing. Its own doc comment names Abort/nanoflow and I still missed it, because I was reading it as \"the safe choice\" rather than \"the choice that preserves THIS call site's prior value\". **`go test ./...` does not run this suite** \u2014 the mx-check round trips are behind `-tags integration` (`make test-integration`, ~30 min), so a green unit suite says nothing about whether mxbuild still accepts what mxcli writes; run it before pushing anything that touches a serialized default. Regression test `TestAuthorOnError_NanoflowKeepsAbortWithoutAClause` (control: the same statements in a microflow must NOT become Abort) \u2014 a unit test, so it catches this in seconds instead of 30 minutes. A second, quieter gap in the same change: `getErrorHandling` in nanoflow_validation.go gates the walk that looks for disallowed actions INSIDE a handler body, so the eight new statements had to be added there too or a Java action nested in `declare \u2026 on error { \u2026 }` goes unreported (test carries a `commit` control, since that entry point is `validateNanoflowBody`, NOT the exported `ValidateNanoflowBody`, which is a different check)", "refs": ["#1078"], "ce": ["CE6035"]} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "In a NANOFLOW, `log ... on error continue|rollback|{ \u2026 }`, and a custom handler on change/show page/close page/show message/validation feedback, are written by mxcli and rejected by mxbuild as **CE6035 \"Error handling type is not supported\"**. `mxcli check` passed. Reachable for the first time via mendixlabs/mxcli#1078, which gave those statements an onErrorClause", "cause": "No rule covered it. MDL076 runs on the microflow validator, which has no flow flavour, so it cannot express \"CE6035 in a nanoflow but fine in a microflow\" \u2014 and every one of these six IS fine in a microflow with a custom handler", "file": "`mdl/executor/nanoflow_validation.go` (`checkNanoflowErrorHandling`, `nanoflowErrorHandlingUnsupported`, wired into `validateNanoflowStatements`)", "insight": "**A nanoflow accepts error handling on almost nothing**: measured on 11.14.0, only the two VARIABLE activities (create-variable, change-variable) take a clause; the other six are CE6035 whichever form is written, because a nanoflow activity's only accepted value is Abort \u2014 the no-clause default, which no MDL syntax writes. So the rule refuses the CLAUSE, not one spelling. The split is by ACTIVITY, not by client-side/server-side: `show message` is as client-side as it gets and still refuses one, while `declare` accepts it. Same permissive pair as `continue` in a microflow, which is the only pattern visible across both tables. **Two validators, two entry points, easy to wire to the wrong one**: `validateNanoflowBody` (unexported) is the disallowed-action walk that exec runs; `ValidateNanoflowBody` (exported) is the variable/semantic check that `check` runs. A test targeting the exported one passes vacuously \u2014 caught only because the test carried a `commit` control that also failed. Note the pre-existing consequence: every nanoflow restriction, including the 22 disallowed action types, is enforced at EXEC and not by `mxcli check` (even with `--references`) \u2014 a check/exec divergence worth closing on its own, but not in a bug fix, since it would newly reject scripts that pass check today", "refs": ["#1078"], "ce": ["CE6035"]} diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 54925e55f4..6574cf5ee0 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -659,6 +659,11 @@ close page on error { return; }; - **The list-operation and aggregate forms of `set`** (`$x = head($l)`, `$n = count($l)`) have no error handling in Mendix at all — **MDL077**. +**In a nanoflow, almost none of them take a clause at all.** `change`, `log`, +`show page`, `close page`, `show message` and `validation feedback` are CE6035 +there whichever form is written; only `declare` and `set` accept one. See +`write-nanoflows`. + **End the handler.** A handler body that does not finish with `return` or `throw` merges back into the main flow, so a variable created *after* the merge point is out of scope on the error path — CE0108, which Studio Pro reports for the same diff --git a/.claude/skills/mendix/write-nanoflows/SKILL.md b/.claude/skills/mendix/write-nanoflows/SKILL.md index 88bb2af386..5235afecb1 100644 --- a/.claude/skills/mendix/write-nanoflows/SKILL.md +++ b/.claude/skills/mendix/write-nanoflows/SKILL.md @@ -550,6 +550,23 @@ For per-action error handling without CONTINUE: $Result = CALL NANOFLOW Sales.NAV_Risky () ON ERROR ROLLBACK; ``` +### Most activities take NO error handling in a nanoflow + +An `ON ERROR` clause of **any** form is rejected on these six, with +**CE6035** "Error handling type is not supported" — measured on Mendix 11.14.0: + +| Refused in a nanoflow | Accepted | +|---|---| +| `CHANGE`, `LOG`, `SHOW PAGE`, `CLOSE PAGE`, `SHOW MESSAGE`, `VALIDATION FEEDBACK` | `DECLARE`, `SET` (the two *variable* activities) | + +mxcli refuses the clause rather than writing a nanoflow mxbuild rejects. The +split is by activity, not by "client-side vs server-side" — `SHOW MESSAGE` is as +client-side as it gets and still refuses one. + +A nanoflow activity **aborts the flow** on error by default (a nanoflow has no +transaction to roll back), which is why there is nothing to configure. That +default is also why writing `Rollback` there is an error and not a no-op. + ## Security (GRANT/REVOKE) ```mdl diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 064ad8a52a..a182490f5d 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -138,6 +138,11 @@ func init() { "-- The list-operation and aggregate forms of SET ($x = head($l),\n" + "-- $n = count($l)) have no error handling in Mendix at all -> MDL077.\n" + "--\n" + + "-- IN A NANOFLOW only DECLARE and SET take a clause at all. CHANGE, LOG,\n" + + "-- SHOW PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION FEEDBACK are CE6035\n" + + "-- there in EVERY form, and are refused: a nanoflow activity aborts the\n" + + "-- flow on error by default and has no transaction to roll back.\n" + + "--\n" + "-- A handler that does NOT end in RETURN/THROW merges back into the main\n" + "-- flow, so a variable created after the merge is out of scope on the error\n" + "-- path (CE0108). End the handler, or expect that.", diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 152a9a517e..e4037ebb5b 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -553,7 +553,7 @@ it is for pages. | Execute DB query | `$Result = execute database query Module.Conn.Query;` | 3-part name; supports DYNAMIC, params, CONNECTION override | | Import mapping | `[$Var =] import from mapping Module.IMM($SourceVar) [all\|first\|limit [offset ]];` | Apply import mapping to string variable. Trailing clause is Studio Pro's Range; omitted = infer from the mapping's root. `first` binds one OBJECT (`limit 1` is a one-element LIST). Mendix rejects `offset` on a non-list mapping (CE6100) | | Export mapping | `$Var = export to mapping Module.EMM($EntityVar);` | Apply export mapping to entity, returns string | -| Error handling | `... on error continue\|rollback\|{ handler }\|without rollback { handler };` | Goes on the activity that may fail — including `declare`, `set`, `change`, `log`, `show page`, `close page`, `show message` and `validation feedback`, which gained it in mendixlabs/mxcli#1078 so a Studio Pro handler survives DESCRIBE. `on error continue` is refused (MDL076) where Mendix raises CE6035: create, change, commit, log, show page, close page, show message, validation feedback — a custom `{ handler }` is accepted on all of them. The list-operation and aggregate forms of `set` have no error handling at all (MDL077). Not supported on EXECUTE DATABASE QUERY. A handler that does not end in `return`/`throw` merges back into the main flow, so a later variable is out of scope on the error path (CE0108) | +| Error handling | `... on error continue\|rollback\|{ handler }\|without rollback { handler };` | Goes on the activity that may fail — including `declare`, `set`, `change`, `log`, `show page`, `close page`, `show message` and `validation feedback`, which gained it in mendixlabs/mxcli#1078 so a Studio Pro handler survives DESCRIBE. `on error continue` is refused (MDL076) where Mendix raises CE6035: create, change, commit, log, show page, close page, show message, validation feedback — a custom `{ handler }` is accepted on all of them. The list-operation and aggregate forms of `set` have no error handling at all (MDL077). Not supported on EXECUTE DATABASE QUERY. **In a nanoflow** only `declare` and `set` take a clause at all — `change`, `log`, `show page`, `close page`, `show message` and `validation feedback` are CE6035 there in every form, and are refused. A handler that does not end in `return`/`throw` merges back into the main flow, so a later variable is out of scope on the error path (CE0108) | **Activity defaults.** An omitted modifier always means Mendix's own default, so a bare MDL statement produces the same activity as dragging a fresh one onto the diff --git a/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl b/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl index 85b844b1b6..4e69224c5f 100644 --- a/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl +++ b/mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl @@ -69,6 +69,14 @@ -- Note create-VARIABLE and change-VARIABLE accept Continue while change-OBJECT -- does not. No rule of thumb predicts that; it was measured. -- +-- IN A NANOFLOW the answer is different again, and stricter: only `declare` and +-- `set` take a clause AT ALL. `change`, `log`, `show page`, `close page`, +-- `show message` and `validation feedback` are CE6035 there in every form, so +-- mxcli refuses the clause rather than writing a nanoflow mxbuild rejects. +-- A nanoflow activity aborts the flow on error by default — there is no +-- transaction to roll back, which is also why writing `Rollback` there is an +-- error rather than a no-op. Everything below is a MICROFLOW for that reason. +-- -- A NON-terminating handler is a different matter and is NOT an mxcli bug: when -- the handler body falls through, the branch merges back into the main flow, and -- a variable defined after the merge point is out of scope on the error path — diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index 1dfb0b6f30..e9a1eee3b5 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -34,8 +34,9 @@ func (fb *flowBuilder) addCreateVariableAction(s *ast.DeclareStmt) model.ID { activityX := fb.posX action := µflows.CreateVariableAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), VariableName: s.Variable, DataType: convertASTToMicroflowDataType(declType, nil), InitialValue: fb.exprToString(s.InitialValue), @@ -74,8 +75,9 @@ func (fb *flowBuilder) addChangeVariableAction(s *ast.MfSetStmt) model.ID { activityX := fb.posX action := µflows.ChangeVariableAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), VariableName: s.Target, Value: fb.exprToString(s.Value), } @@ -331,8 +333,9 @@ func (fb *flowBuilder) addChangeObjectAction(s *ast.ChangeObjectStmt) model.ID { activityX := fb.posX action := µflows.ChangeObjectAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), ChangeVariable: s.Variable, Commit: commitTypeOf(s.Commit), RefreshInClient: s.RefreshInClient || len(s.Changes) == 0, diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index daffbeb5a7..0665a606b0 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -80,8 +80,9 @@ func (fb *flowBuilder) addLogMessageAction(s *ast.LogStmt) model.ID { activityX := fb.posX action := µflows.LogMessageAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), LogLevel: logLevel, LogNodeName: logNodeName, MessageTemplate: &model.Text{ @@ -954,8 +955,9 @@ func (fb *flowBuilder) addShowPageAction(s *ast.ShowPageStmt) model.ID { activityX := fb.posX action := µflows.ShowPageAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), PageName: pageQN, // BY_NAME_REFERENCE - qualified name string PageSettings: pageSettings, PageParameterMappings: mappings, @@ -1054,8 +1056,9 @@ func (fb *flowBuilder) addShowMessageAction(s *ast.ShowMessageStmt) model.ID { activityX := fb.posX action := µflows.ShowMessageAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), Template: template, Type: msgType, TemplateParameters: templateParams, @@ -1159,8 +1162,9 @@ func (fb *flowBuilder) addClosePageAction(s *ast.ClosePageStmt) model.ID { activityX := fb.posX action := µflows.ClosePageAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), NumberOfPages: numPages, } @@ -1281,8 +1285,9 @@ func (fb *flowBuilder) addValidationFeedbackAction(s *ast.ValidationFeedbackStmt activityX := fb.posX action := µflows.ValidationFeedbackAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ErrorHandlingType: explicitErrorHandling(fb, s.ErrorHandling), + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + // fb.ehType, not explicitErrorHandling — see ehType's doc comment. + ErrorHandlingType: fb.ehType(s.ErrorHandling), ObjectVariable: varName, AttributeName: attributeName, AssociationName: associationName, diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 79e032161e..5079368667 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -32,6 +32,23 @@ func convertErrorHandlingType(eh *ast.ErrorHandlingClause) microflows.ErrorHandl // ehType returns the error handling type for an activity in this flow context. // Nanoflows default to "Abort" because they have no transactions; microflows // default to "Rollback". An explicit ON ERROR clause always overrides the default. +// +// Most builders want THIS, not explicitErrorHandling below, and the two are not +// interchangeable — picking the wrong one is a silent CE6035. Which is right +// depends entirely on what the call site did before: +// +// - A builder that already supplied a default here (every create/change/log/ +// page/message/validation activity) must keep using ehType. Returning empty +// discards the flow flavour, and the writer's literal "Rollback" is CE6035 on +// every un-annotated activity in a NANOFLOW, whose default is Abort. That is +// mendixlabs/mxcli#1078's regression: green unit suite, 11 errors under +// `make test-integration`. +// - Retrieve and Delete use explicitErrorHandling because their writers emitted +// a hardcoded "Rollback" that those two actions accept in every flow flavour, +// so empty is a no-op there. +// +// Same helper pair, opposite correct answer. Ask what the old expression returned +// in EVERY context before replacing it, not just the one under test. func (fb *flowBuilder) ehType(eh *ast.ErrorHandlingClause) microflows.ErrorHandlingType { if fb.isNanoflow && eh == nil { return microflows.ErrorHandlingTypeAbort diff --git a/mdl/executor/microflow_error_handler_authoring_test.go b/mdl/executor/microflow_error_handler_authoring_test.go index c57a2b2c6b..2a9442a387 100644 --- a/mdl/executor/microflow_error_handler_authoring_test.go +++ b/mdl/executor/microflow_error_handler_authoring_test.go @@ -109,10 +109,19 @@ func TestAuthorOnError_EightStatementsThatCouldNotCarryTheClause(t *testing.T) { } } -// Control for the table above. Without the clause the action must carry NO -// error-handling type: writing one would make DESCRIBE render `on error` on an -// activity the author never put one on, which is #840 in reverse. -func TestAuthorOnError_AbsentClauseLeavesTheActionUnset(t *testing.T) { +// Control for the table above. Without a clause the activity must keep the +// flow's own default and gain no error branch. +// +// The default is Rollback in a microflow (Abort in a nanoflow — see +// TestAuthorOnError_NanoflowKeepsAbortWithoutAClause), NOT the empty string. +// Empty was this test's original expectation and it was wrong in the direction +// that matters: it is what the writer turns into a literal "Rollback" +// regardless of flow flavour, which is CE6035 inside a nanoflow. +// +// Rollback is also exactly the value DESCRIBE must NOT render a suffix for +// (#840), so the two halves of the round trip agree: store the default, print +// nothing. +func TestAuthorOnError_AbsentClauseKeepsTheFlowDefault(t *testing.T) { for _, body := range []string{ "declare $name String = 'NameValue';", "log info node 'B' 'hi';", @@ -124,11 +133,15 @@ func TestAuthorOnError_AbsentClauseLeavesTheActionUnset(t *testing.T) { t.Fatalf("%s: no action activity", body) } for _, errType := range types { - if errType != "" { - t.Errorf("%s: ErrorHandlingType = %q, want empty — no clause was written", - body, errType) + if errType != microflows.ErrorHandlingTypeRollback { + t.Errorf("%s: ErrorHandlingType = %q, want %q (the microflow default)", + body, errType, microflows.ErrorHandlingTypeRollback) } } + // The part that actually gates DESCRIBE: no custom handler, no branch. + if n := countErrorHandling(fb, microflows.ErrorHandlingTypeCustom); n != 0 { + t.Errorf("%s: %d activities came back Custom with no clause written", body, n) + } for _, f := range fb.flows { if f.IsErrorHandler { t.Errorf("%s: an error-handler flow was created with no clause", body) @@ -195,3 +208,163 @@ func checkMicroflowBodyForTest(t *testing.T, body string) string { } return b.String() } + +// The regression #1078 shipped and CI caught: a NANOFLOW activity with no +// clause must keep Abort, not fall through to the writer's "Rollback". +// +// Eight builders here previously set fb.ehType(nil) — context-dependent, and +// Abort inside a nanoflow. Switching them to explicitErrorHandling (which +// returns empty for "no clause") made the writer's literal "Rollback" apply +// instead, and mxbuild reports CE6035 "Error handling type is not supported" on +// every un-annotated change/log/close-page/validation-feedback activity in a +// nanoflow. Retrieve and Delete legitimately use explicitErrorHandling — their +// writers emitted a hardcoded "Rollback" those two actions accept everywhere — +// so the helper is right there and wrong here. +// +// go test ./... never saw it: the failure is in the mx-check integration suite +// (-tags integration), not the unit suite. +func TestAuthorOnError_NanoflowKeepsAbortWithoutAClause(t *testing.T) { + for _, tc := range []struct{ name, body string }{ + {"change object", "change $Car (Brand = 'Opel');"}, + {"log", "log info node 'B' 'hi';"}, + {"close page", "close page;"}, + {"show message", "show message 'hi';"}, + {"validation feedback", "validation feedback $Car/Brand message 'bad';"}, + {"declare", "declare $name String = 'v';"}, + {"set", "declare $name String = 'v';\n$name = 'w';"}, + } { + t.Run(tc.name, func(t *testing.T) { + fb := buildNanoflowFromMDL(t, tc.body) + for _, got := range actionErrorHandlingTypes(fb) { + if got != microflows.ErrorHandlingTypeAbort { + t.Errorf("nanoflow activity carries ErrorHandlingType %q, want %q — "+ + "an empty value falls through to the writer's \"Rollback\", which "+ + "mxbuild rejects as CE6035 in a nanoflow", + got, microflows.ErrorHandlingTypeAbort) + } + } + }) + } + + // Control: the same statements in a MICROFLOW must not become Abort. + fb := buildFlowFromMDL(t, "log info node 'B' 'hi';") + for _, got := range actionErrorHandlingTypes(fb) { + if got == microflows.ErrorHandlingTypeAbort { + t.Errorf("microflow activity got Abort, which is CE6035 outside a nanoflow") + } + } +} + +// buildNanoflowFromMDL is buildFlowFromMDL with the nanoflow flag set, which is +// the only thing that changes the no-clause default. +func buildNanoflowFromMDL(t *testing.T, body string) *flowBuilder { + t.Helper() + prog, errs := visitor.Build("create microflow M.ACT_T()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatalf("parsing:\n%s\nerrors: %v", body, errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + fb := &flowBuilder{ + posX: 100, posY: 100, spacing: HorizontalSpacing, isNanoflow: true, + varTypes: map[string]string{}, declaredVars: map[string]string{}, + } + fb.buildFlowGraph(mf.Body, nil) + return fb +} + +// A nanoflow's error-handler BODY is walked for actions nanoflows cannot run. +// The eight statements #1078 opened up had to be added to getErrorHandling for +// that walk to reach them — otherwise a Java action nested in +// `declare … on error { … }` is accepted and fails only at build time. +func TestNanoflow_HandlerBodyOfNewStatementsIsValidated(t *testing.T) { + // Control first: a statement that could ALREADY carry the clause. If this + // stops reporting, the walk itself broke and the rows below prove nothing. + if errs := nanoflowErrorsFor(t, + "commit $Obj on error {\n call java action M.SomeJava();\n};"); len(errs) == 0 { + t.Fatal("control failed: a Java action inside a commit handler was accepted, " + + "so this test cannot detect anything") + } + + for _, tc := range []struct{ name, body string }{ + {"declare", "declare $n String = 'v' on error {\n call java action M.SomeJava();\n};"}, + {"set", "declare $n String = 'v';\n$n = 'w' on error {\n call java action M.SomeJava();\n};"}, + {"change object", "change $Car (Brand = 'x') on error {\n call java action M.SomeJava();\n};"}, + {"log", "log info node 'B' 'hi' on error {\n call java action M.SomeJava();\n};"}, + {"show page", "show page M.Home on error {\n call java action M.SomeJava();\n};"}, + {"close page", "close page on error {\n call java action M.SomeJava();\n};"}, + {"show message", "show message 'hi' on error {\n call java action M.SomeJava();\n};"}, + {"validation feedback", "validation feedback $Car/Brand message 'b' on error {\n call java action M.SomeJava();\n};"}, + } { + t.Run(tc.name, func(t *testing.T) { + if errs := nanoflowErrorsFor(t, tc.body); len(errs) == 0 { + t.Errorf("a Java action inside this handler was accepted — the nanoflow "+ + "walk does not reach %s's error body", tc.name) + } + }) + } +} + +// nanoflowErrorsFor parses a nanoflow body and returns the nanoflow-specific +// validation errors. +func nanoflowErrorsFor(t *testing.T, body string) []string { + t.Helper() + prog, errs := visitor.Build("create nanoflow M.NF_T()\nbegin\n" + body + "\nreturn;\nend;") + if len(errs) > 0 { + t.Fatalf("parsing:\n%s\nerrors: %v", body, errs) + } + // validateNanoflowBody, not the exported ValidateNanoflowBody: the latter + // runs the variable/semantic checks, the former is the disallowed-action walk + // this test is about. + return validateNanoflowBody(prog.Statements[0].(*ast.CreateNanoflowStmt).Body) +} + +// A nanoflow accepts error handling on almost none of the eight statements +// #1078 opened up: measured on 11.14.0, only the two VARIABLE activities take a +// clause, and the other six are CE6035 whichever form is written. Refused at +// exec rather than written into a nanoflow mxbuild rejects. +func TestNanoflow_RefusesErrorHandlingWhereMendixRejectsIt(t *testing.T) { + for _, tc := range []struct{ name, body string }{ + {"change object", "change $Car (Brand = 'x') on error { show message 'e'; };"}, + {"log", "log info node 'B' 'hi' on error { show message 'e'; };"}, + {"show page", "show page M.Home on error { show message 'e'; };"}, + {"close page", "close page on error { show message 'e'; };"}, + {"show message", "show message 'hi' on error { close page; };"}, + {"validation feedback", "validation feedback $Car/Brand message 'b' on error { close page; };"}, + // Log is the activity measured in all three forms; all three are CE6035, + // which is why the rule refuses the clause rather than one spelling. + {"log continue", "log info node 'B' 'hi' on error continue;"}, + {"log rollback", "log info node 'B' 'hi' on error rollback;"}, + } { + t.Run(tc.name, func(t *testing.T) { + errs := nanoflowErrorsFor(t, tc.body) + if !containsSubstringAny(errs, "on error") { + t.Errorf("accepted in a nanoflow, but mxbuild reports CE6035: %v", errs) + } + }) + } + + // Control 1: the permissive pair. Refusing these would reject nanoflows that + // build today — measured, both accept a custom handler on 11.14.0. + for _, body := range []string{ + "declare $n String = 'v' on error { show message 'e'; };", + "declare $n String = 'v';\n$n = 'w' on error { show message 'e'; };", + } { + if errs := nanoflowErrorsFor(t, body); containsSubstringAny(errs, "on error") { + t.Errorf("a variable activity was refused, but Mendix accepts it: %v", errs) + } + } + + // Control 2: no clause at all must never be refused. + if errs := nanoflowErrorsFor(t, "log info node 'B' 'hi';"); containsSubstringAny(errs, "on error") { + t.Errorf("an activity with no clause was refused: %v", errs) + } +} + +func containsSubstringAny(errs []string, want string) bool { + for _, e := range errs { + if strings.Contains(e, want) { + return true + } + } + return false +} diff --git a/mdl/executor/nanoflow_validation.go b/mdl/executor/nanoflow_validation.go index c0f3cb4786..9b80ef7a09 100644 --- a/mdl/executor/nanoflow_validation.go +++ b/mdl/executor/nanoflow_validation.go @@ -24,6 +24,9 @@ func validateNanoflowStatements(stmts []ast.MicroflowStatement, errors *[]string *errors = append(*errors, reason) continue } + if reason := checkNanoflowErrorHandling(stmt); reason != "" { + *errors = append(*errors, reason) + } // Recurse into compound statements switch s := stmt.(type) { case *ast.IfStmt: @@ -113,6 +116,26 @@ func getErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClause { return s.ErrorHandling case *ast.CallJavaScriptActionStmt: return s.ErrorHandling + // The eight statements mendixlabs/mxcli#1078 gave an onErrorClause. None is + // on the denylist above, so all eight are reachable in a nanoflow — and + // without them here their handler BODIES are never walked, so a Java action + // or REST call nested inside `declare … on error { … }` would go unreported. + case *ast.DeclareStmt: + return s.ErrorHandling + case *ast.MfSetStmt: + return s.ErrorHandling + case *ast.ChangeObjectStmt: + return s.ErrorHandling + case *ast.LogStmt: + return s.ErrorHandling + case *ast.ShowPageStmt: + return s.ErrorHandling + case *ast.ClosePageStmt: + return s.ErrorHandling + case *ast.ShowMessageStmt: + return s.ErrorHandling + case *ast.ValidationFeedbackStmt: + return s.ErrorHandling } return nil } @@ -152,3 +175,81 @@ func validateNanoflow(name string, body []ast.MicroflowStatement, retType *ast.M } return errMsg.String() } + +// nanoflowErrorHandlingUnsupported names the activities that accept NO error +// handling at all inside a nanoflow, by the caption mxbuild uses. +// +// MEASURED on Mendix 11.14.0, one nanoflow per cell, not inferred. A dash is a +// cell that was NOT measured, not one that passed — every activity listed in the +// map below has at least one measured CE6035, and none has a measured pass: +// +// activity (in a NANOFLOW) continue rollback custom { } +// CreateVariable (declare) ok - ok +// ChangeVariable (set) - - ok +// ChangeObject - CE6035 CE6035 +// Log CE6035 CE6035 CE6035 +// ShowPage - - CE6035 +// ClosePage - CE6035 CE6035 +// ShowMessage - - CE6035 +// ValidationFeedback - CE6035 CE6035 +// +// The rollback column comes from #1078's own regression: writing "Rollback" +// instead of the nanoflow default Abort was CE6035 on exactly ChangeObject, +// ClosePage and ValidationFeedback in the doctype scripts. Log is the one +// activity measured in all three forms, and rejects all three — which is why the +// rule refuses any clause rather than one form: the accepted value is Abort, and +// no MDL syntax writes it. +// +// The two VARIABLE activities are the permissive pair here, exactly as they are +// for `continue` in a microflow (see continueUnsupportedOn) — the split is by +// activity, not by "client-side vs server-side". +// +// This is a nanoflow-only rule and cannot live in MDL076: that one runs on the +// microflow validator, which has no flow flavour. Reaching this at all is new — +// mendixlabs/mxcli#1078 gave these six statements an onErrorClause so a Studio +// Pro error handler could survive DESCRIBE, and a nanoflow accepts none of them. +var nanoflowErrorHandlingUnsupported = map[string]string{ + "change": "Change object activity", + "log": "Log message activity", + "show page": "Show page activity", + "close page": "Close page activity", + "show message": "Show message activity", + "validation feedback": "Validation feedback activity", +} + +// checkNanoflowErrorHandling reports an ON ERROR clause on a nanoflow activity +// that cannot carry one. +// +// Refused rather than dropped or downgraded: the alternatives are a nanoflow +// mxbuild rejects (CE6035) or one that silently does something the script does +// not say. `mxcli check` passed the rejected form until this rule existed. +func checkNanoflowErrorHandling(stmt ast.MicroflowStatement) string { + if getErrorHandling(stmt) == nil { + return "" + } + var keyword string + switch stmt.(type) { + case *ast.ChangeObjectStmt: + keyword = "change" + case *ast.LogStmt: + keyword = "log" + case *ast.ShowPageStmt: + keyword = "show page" + case *ast.ClosePageStmt: + keyword = "close page" + case *ast.ShowMessageStmt: + keyword = "show message" + case *ast.ValidationFeedbackStmt: + keyword = "validation feedback" + default: + // declare and set are deliberately absent: both variable activities accept + // every form in a nanoflow. So do the statements that could already carry + // the clause (commit, create, retrieve, the calls) — unmeasured here, and + // refusing them would reject nanoflows that build today. + return "" + } + return "`" + keyword + " ... on error` is not supported in a nanoflow — Mendix rejects " + + "error handling on a " + nanoflowErrorHandlingUnsupported[keyword] + + " there with CE6035 \"Error handling type is not supported\". Drop the clause " + + "(a nanoflow activity aborts the flow on error by default)" +} From eb6ca393a1618ff0c46d4dc2ec1622577e0a2c87 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 07:43:41 +0000 Subject: [PATCH 16/19] build(deps-dev): Bump svgo Bumps [svgo](https://github.com/svg/svgo) from 2.8.3 to 2.8.4. - [Release notes](https://github.com/svg/svgo/releases) - [Commits](https://github.com/svg/svgo/compare/v2.8.3...v2.8.4) --- updated-dependencies: - dependency-name: svgo dependency-version: 2.8.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] --- .../packs/mendix-vega-charts/widget/package-lock.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json b/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json index 40b6bce344..1dfec496bf 100644 --- a/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json +++ b/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json @@ -20875,9 +20875,9 @@ } }, "node_modules/svgo": { - "version": "2.8.3", - "resolved": "https://registry.npmjs.org/svgo/-/svgo-2.8.3.tgz", - "integrity": "sha512-5EZD0pafXX6PphdwOGCiVLDSaV1xyuQao2blHajHLsPxr07q4mmEjdtXEWgG07ae2mIz8Ex2CDXNCTiXhy3Khw==", + "version": "2.8.4", + "resolved": "https://registry.npmjs.org/svgo/-/svgo-2.8.4.tgz", + "integrity": "sha512-2GJ4h3rl13qYTdwllaK6QlL8tG+UrM8626V2Ylcd/yBUv2Y/EwLsYisVV6UDCRmlk+C74G8nMpxJCk0RWPdDCw==", "dev": true, "license": "MIT", "dependencies": { From cd7dca80a1b00c86e70d4a1f2b1a0a07ffe372f5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 10 Sep 2026 07:43:42 +0000 Subject: [PATCH 17/19] build(deps): Bump js-yaml Bumps and [js-yaml](https://github.com/nodeca/js-yaml). These dependencies needed to be updated together. Updates `js-yaml` from 4.3.1 to 4.3.2 - [Changelog](https://github.com/nodeca/js-yaml/blob/4.3.2/CHANGELOG.md) - [Commits](https://github.com/nodeca/js-yaml/compare/4.3.1...4.3.2) Updates `js-yaml` from 3.15.1 to 3.15.2 - [Changelog](https://github.com/nodeca/js-yaml/blob/4.3.2/CHANGELOG.md) - [Commits](https://github.com/nodeca/js-yaml/compare/4.3.1...4.3.2) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.3.2 dependency-type: indirect - dependency-name: js-yaml dependency-version: 3.15.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] --- .../mendix-vega-charts/widget/package-lock.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json b/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json index 40b6bce344..2376edc92c 100644 --- a/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json +++ b/.claude/skills/packs/mendix-vega-charts/widget/package-lock.json @@ -2622,9 +2622,9 @@ } }, "node_modules/@eslint/eslintrc/node_modules/js-yaml": { - "version": "4.3.1", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.1.tgz", - "integrity": "sha512-CY6crGq313MX8GkwvB7tzgp99vjQxY1++5y10/BKN/GUfHqWaOGQMNZkBvqSzsZKWk/ijwHlWzzkLulsGHhjWQ==", + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.2.tgz", + "integrity": "sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==", "dev": true, "funding": [ { @@ -15844,9 +15844,9 @@ "license": "MIT" }, "node_modules/js-yaml": { - "version": "3.15.1", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.15.1.tgz", - "integrity": "sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag==", + "version": "3.15.2", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.15.2.tgz", + "integrity": "sha512-6EuL879VkRA+1Cz578mKMiKvjPNEuk6+r1JaFzoSWejZmtf7xWbIyw1e3KkxlkzTIt9Taw6JBhEppG7utc1P+w==", "dev": true, "license": "MIT", "dependencies": { From ec7ef1e61707cee0bd8a5cf21995510dce70ca7c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 12:56:48 +0000 Subject: [PATCH 18/19] fix(catalog): never write a narrower catalog over a richer cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mxcli check -p` emptied the catalog's source index (mendixlabs/mxcli#1081). Measured on testdata/expr-checker: a source-mode cache of 2,191,360 bytes with 106 SOURCE rows became 1,253,376 bytes with 0 after one check — and not only source, but REFS 271→0, PERMISSIONS 162→0, STRINGS 1496→0, XPATH_EXPRESSIONS 5→0, so show references/callers/impact, search and show languages answered "requires refresh catalog full source" instead of answering. The cache records the mode it was built in and the modes nest (source ⊃ full ⊃ fast). check's expression-type tier asks for a fast catalog, which a source cache satisfies — so while the cache was valid nothing was lost. The loss began the moment it went stale, which in practice is any project save, `mxcli exec` or branch switch between two commands: buildCatalog then wrote its fast rebuild over the cache unconditionally. Guarded at the single save site, not at check: `show structure`, `describe`, `show catalog tables` and plain `refresh catalog` reach the same seam and reproduce it identically. The reasoning was already in the file — `refresh catalog communities` carries a comment describing this exact hazard — applied as a local branch instead of at the choke point, which is how every other caller kept the bug. Two clauses go with the guard, each with its own control, because the guard alone trades a data-loss bug for a performance one: - A rebuild that pays the full-mode cost anyway rebuilds at the cached level, so the cache comes back current instead of being refused and rebuilt on every invocation. Stubbing this leaves mode and row count correct; only asserting the cache is valid afterwards catches it. - REFRESH CATALOG refreshes at the cached level. "Refresh" means bring what this project has up to date, and there is no syntax for lowering the level, so the guard alone would turn a bare refresh into a no-op on disk. A fast consumer is deliberately not upgraded: it is the hot path and the cache goes stale on every project save, so upgrading would put a source reindex inside the exec → check → exec loop. It builds narrow in memory instead, +0.12s per invocation on the fixture (0.48s cached, 0.60s rebuilt), replacing a fast/full/source thrash in which the last builder destroyed the others' tables. The cache's mode is now sticky; delete .mxcli/catalog.db to drop to a cheaper one, documented in the CLI help and the refresh-catalog command. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01L45JFiJ6y58ftg2zWVEq7h --- .claude/commands/mendix/refresh-catalog.md | 6 + .../fix-issue/findings/mdl-executor.jsonl | 1 + CHANGELOG.md | 8 + mdl/executor/catalog_cache_mode_test.go | 220 ++++++++++++++++++ mdl/executor/cmd_catalog.go | 113 ++++++++- mdl/executor/cmd_misc.go | 5 +- 6 files changed, 345 insertions(+), 8 deletions(-) create mode 100644 mdl/executor/catalog_cache_mode_test.go diff --git a/.claude/commands/mendix/refresh-catalog.md b/.claude/commands/mendix/refresh-catalog.md index 58a993b9b0..7bafed0473 100644 --- a/.claude/commands/mendix/refresh-catalog.md +++ b/.claude/commands/mendix/refresh-catalog.md @@ -31,6 +31,12 @@ REFRESH CATALOG FULL FORCE; - Use FULL mode before using SEARCH command - Use SOURCE mode for searching MDL definitions +The cache's mode is sticky. Once a project has a FULL or SOURCE cache, a plain +`REFRESH CATALOG` rebuilds at that level rather than dropping to FAST, and a +command that needs less (`mxcli check -p`, `show structure`, `describe`) never +replaces it with a narrower one. To go back to a cheaper level, delete +`.mxcli/catalog.db` and refresh. + ## Example Queries After Refresh ```sql diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index b810fb205f..68af13ec09 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -558,3 +558,4 @@ {"area": "mdl/executor", "date": "2026-09-07", "symptom": "`describe workflow` emitted MDL that `mxcli check` refused: 6 of the 14 workflows in the 9 demo apps in mx-test-projects/ failed describe -> check. Two rules fired: MDL-WF03 on decision outcomes like 'FactoryManagement.ENUM_InvestigationType.Engineering', and MDL-WF05 on `jump to decision1` / `jump to split1`.", "cause": "Two independent defects behind one symptom. (1) wfOutcomeIdentRe required a BARE identifier, but every Workflows$EnumerationValueConditionOutcome in the corpus stores the QUALIFIED form Module.Enum.Value (7 of 7 non-empty) — the rule was written to catch free text like 'Confirmed closed' and rejecting the dot was collateral, so the describer was right and the validator wrong. (2) MDL had a name slot only on `user task`; every other builder did act.Name = act.Caption, while Mendix resolves JumpToActivity.TargetActivity by activity NAME and Studio Pro names activities by type and ordinal (decision1, split1, callMicroflow1, userTask1, waitForNotification1, timer1) with no relation to the caption. The stored name had nowhere to be emitted to.", "file": "mdl/executor/validate_workflow.go, mdl/grammar/domains/MDLWorkflow.g4, mdl/executor/cmd_workflows.go, mdl/executor/cmd_workflows_write.go", "insight": "The second half was NOT a describer bug, which is what it looked like at first: the grammar had no name slot, so the fix ran grammar -> AST -> visitor -> builder -> describer. Two measurements settled the design and neither was guessable from the code. Studio Pro's stored outcome value decided which side of defect 1 to change — changing the describer to emit the last segment would have 'fixed' the check and written a document unlike every real one. And TargetActivity turned out to hold a NAME STRING, not an ID pointer, which is why a lost name degrades to a jump-to-itself and surfaces as CE6681 'not possible to jump to end activities or jump-to activities' — an error naming a different fault entirely. Emit the name only when it is not derivable (name != caption and != sanitizeActivityName(caption); for call activities, != the called document's short name), so mxcli-authored workflows describe unchanged and the clause appears exactly where it carries information. The control is what makes this provable: the pre-fix binary, built from HEAD~1 in a throwaway worktree, reproduces 6/14 failing where the fixed one is 0/14, and the round-tripped document was read back to confirm decision1..3 / split1 / callMicroflow1..6 landed and both jump targets resolve — mx check at 0 errors alone would NOT have shown that, since a workflow with a jump to itself is perfectly valid.", "issue": "ako/mxcli#408"} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "`mxcli check --references` reported EVERY enumeration as missing — `attribute 'CriticalPathStation': enumeration not found: Approval.StationKey` — while `DESCRIBE ENUMERATION` returned its values, `SHOW ENUMERATIONS` listed it, `exec` wrote the attribute and mxbuild built the project at 0 errors (mendixlabs/mxcli#1071). A pure false negative: the only broken thing was the checker.", "cause": "`enumerationExists` (mdl/executor/helpers.go) matched containers directly — `enum.ContainerID == module.ID` — which only holds for an enumeration sitting in the module ROOT; one inside a FOLDER has the folder as its container. Every other command resolves through the container hierarchy (`h.GetModuleName(h.FindModuleID(e.ContainerID))`), so the reference checker was the only one that could not see inside a folder. Fixed by deferring to `findEnumeration`, deleting the duplicate rather than patching the copy.", "file": "`mdl/executor/helpers.go` (enumerationExists); call sites `mdl/executor/validate.go:447` (CREATE ENTITY) and `:660` (ALTER ENTITY ADD ATTRIBUTE); tests `mdl/executor/validate_enum_folder_test.go`; example `mdl-examples/bug-tests/1071-foldered-enum-references.mdl`", "insight": "This is upstream #976 a second time. That fix corrected DROP's container matching and did NOT sweep for the other callers asking the same question, so the identical bug sat in the reference checker for months — and its own test file already spelled out the class (\"SHOW, DESCRIBE and ALTER all use the container hierarchy... DROP was the one command of the four\"). When a fix is 'this command resolved containers wrongly', grep for every other place that resolves the same containers before closing it; the enumeration existed in TWO implementations and only the interactive one was ever exercised. Two measurement notes: the report read as 'enumerations are never resolved' because the reporter's module keeps them in folders, so the discriminator (root vs foldered, same module, same script) had to be built before anything else made sense; and the blast radius was larger than reported — CREATE ENTITY fails identically and the report only showed ALTER, so the test covers both call sites.", "ce": []} {"area": "mdl/executor", "date": "2026-09-08", "symptom": "A workflow `decision` branching on an enumeration passed `mxcli check` and `mxcli exec`, then mxbuild rejected the project with CE6686 \"The current outcomes of the decision activity do not match the configured expression. Regenerate the outcomes.\" The same on a `call microflow` activity branching on an enumeration-returning microflow (\"...of the call microflow activity do not match the configured microflow\"). Reachable from ALTER WORKFLOW as well: an `INSERT AFTER … decision` without the empty outcome takes a project sitting at 0 errors to 1.", "cause": "Mendix generates a decision's outcome set as one outcome per enumeration value PLUS one with an EMPTY value, and mxbuild compares the stored set against that generated set. Nothing in mxcli knew about the empty outcome: MDL-WF03 validated each outcome NAME and explicitly skipped the empty one (`o.Value == \"\"` continue), and no rule looked at the SET. Studio Pro's own documents carry the extra Workflows$EnumerationValueConditionOutcome with Value ''. And no workflow rule had ever run on an ALTER-introduced activity — ValidateWorkflow is keyed on CreateWorkflowStmt, so every MDL-WF rule was blind to INSERT AFTER / REPLACE ACTIVITY.", "file": "mdl/executor/validate_workflow.go (checkWorkflowEmptyEnumOutcome, MDL-WF06), mdl-examples/bug-tests/wf-enum-decision-empty-outcome{,.fail}.mdl, mdl/executor/validate_program.go (ValidateAlterWorkflow wiring)", "insight": "The severity was the whole question, and intuition had it backwards. 'The empty outcome must be for nullable attributes' is the obvious reading and it is wrong: measured on mxbuild 11.10.0, a decision on an attribute carrying a REQUIRED (not null) validation rule is still CE6686 without `'' -> { }` — so it is an unconditional property of the enumeration TYPE, not of the value, and the rule is an error rather than a warning. Two more measurements shaped the scope and neither was guessable. (1) The same CE fires on a call-microflow activity branching on an enumeration return and clears the same way, so the rule runs at BOTH call sites — the reported bug named only decisions, and stopping there would have left half the class open (the 'probe every sibling' rule). (2) mxbuild wants set EQUALITY, not 'contains empty': a two-value enum with Standard + '' is also 1 error. That half needs the enumeration's definition, so it belongs to the reference pass, not to a syntax-only rule — worth stating in the code so the next person does not read the rule as complete. Classify outcomes with buildConditionOutcome's own switch (True/False -> boolean, Default -> void, everything else -> enum) rather than a second reading of 'looks like an enum value', or the two drift on the first grammar change. The control that proves the rule detects anything is a PAIR of fixtures, not the .fail.mdl alone: `.fail.mdl` only asserts a non-zero exit, which a rule that refused every enumeration decision would also produce — so the identical script WITH the empty outcomes sits next to it and must keep passing (0 errors on mxbuild, measured both ways via exec --no-check). The ALTER half is the 'probe every sibling' rule paying out twice: the reported construct was a CREATE decision, and the same statement shape is reachable through ALTER, where NOTHING was validated. Port only the rules whose verdict is complete in the introduced subtree — MDL-WF06 qualifies, MDL-WF01/WF02 do not (a later `SET ACTIVITY … PAGE` in the same script repairs them) and MDL-WF05 cannot (it resolves jump targets against activities the ALTER statement never sees). The false-positive worry for MDL-WF06 on ALTER — a following `INSERT OUTCOME '' ON decisionN` completing the set — turned out not to exist, and the probe found a separate defect instead: INSERT OUTCOME on a decision writes a Workflows$UserTaskOutcome into a ConditionOutcome list, and the project then fails to LOAD (System.InvalidCastException at UnitContentsLoader.FillProperties), which is the MDL-WF04 class, not a build error.", "refs": ["ako/mxcli#408"], "ce": ["CE6686"], "rules": ["MDL-WF06"]} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "`mxcli check -r` / `check -p` emptied the catalog's source table and shrank .mxcli/catalog.db on disk (mendixlabs/mxcli#1081, reported on 0.21). Measured on testdata/expr-checker: a source-mode cache of 2,191,360 bytes with 106 SOURCE rows became 1,253,376 bytes with 0 after one check. Not only source — REFS 271→0, PERMISSIONS 162→0, STRINGS 1496→0, XPATH_EXPRESSIONS 5→0, so `show references/callers/impact`, `search` and `show languages` all went quiet with a 'requires refresh catalog full source' tip instead of an answer.", "cause": "buildCatalog persisted whatever mode it had just built over .mxcli/catalog.db unconditionally (os.Remove + SaveToFile), with no regard for the mode the existing cache recorded. check's typecheck tier asks ensureCatalog for FAST (all it needs is attribute types, enum cases and microflow return types); while the cache is valid a source cache satisfies fast and nothing is lost, but the moment it is invalid — in practice 'project file modified', i.e. any Studio Pro save, `mxcli exec` or branch switch between two commands — it fell through to a fast build that became the new cache. So `check` was the reporter's entry point, not the defect: `show structure`, `describe`, `show catalog tables` and plain `refresh catalog` all reproduce it through the same seam.", "file": "mdl/executor/cmd_catalog.go (catalogModeRank, cachedCatalogMode, the never-narrow guard in buildCatalog, the upgrade clauses in ensureCatalog and execRefreshCatalogStmt), mdl/executor/catalog_cache_mode_test.go", "insight": "The fix for this exact hazard was already in the file, applied one call site too high. `refresh catalog communities` carries a comment saying 'A rebuild would downgrade a source-mode catalog to full (dropping the source FTS data) … ensureCatalog loads the current full/source cache (preserving its mode)' — the reasoning was correct, was written down, and was implemented as a local branch instead of a guard at the single save site, so every other caller kept the bug. When a comment explains why one caller must not do X, that is evidence X belongs at the choke point. Three clauses, and each needed its own control because the obvious ones subsume each other: (1) never write a narrower catalog over a wider one — stub it and the two check tests fail with the reported symptom; (2) a rebuild that pays the full-mode cost anyway rebuilds at the cached level, else `search` builds full, is refused by (1), and rebuilds full again on every invocation — stubbing this leaves mode and row count CORRECT and is caught only by asserting the cache is VALID afterwards; (3) explicit REFRESH CATALOG refreshes at the cached level, since guard (1) alone turns a bare refresh into a no-op on disk — same trap, same isCacheValid assertion catches it, and the mode/row assertions do not. A fast consumer is deliberately NOT upgraded to source: it is the hot path and the cache goes stale on every project save, so upgrading would put a source reindex inside the exec→check→exec loop. The cost of not persisting it is one in-memory rebuild per invocation between edits, measured at +0.12s on the fixture (0.48s cached vs 0.60s rebuilt) — and it replaces a pre-existing fast/full/source thrash in which the last builder won and destroyed the others' tables. An MDL fixture in mdl-examples/bug-tests/ was deliberately not added: the symptom is a file on disk across two process invocations, which `make check-mdl` (syntax only) cannot express, so it would assert nothing.", "refs": ["mendixlabs/mxcli#1081"], "ce": [], "rules": []} diff --git a/CHANGELOG.md b/CHANGELOG.md index 73673676cd..99a032e7ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`mxcli check -p` emptied the catalog's source index** (mendixlabs/mxcli#1081) — running a check against a project shrank `.mxcli/catalog.db` and left `CATALOG.SOURCE` at zero rows. Measured on `testdata/expr-checker`: a source-mode cache of 2,191,360 bytes with 106 source rows became 1,253,376 bytes with 0. Not just source — `REFS` 271→0, `PERMISSIONS` 162→0, `STRINGS` 1496→0, `XPATH_EXPRESSIONS` 5→0, so `show references`/`callers`/`impact`, `search` and `show languages` answered "requires refresh catalog full source" instead of answering. + + The catalog cache records the mode it was built in, and the modes nest: source ⊃ full ⊃ fast. `check`'s expression-type tier asks for a **fast** catalog — attribute types, enumeration cases, microflow return types — and while the cache is valid a source cache satisfies that, so nothing was lost. The loss began the moment the cache went stale, which in practice means *any* project save, `mxcli exec` or branch switch between two commands: the fast rebuild was then written over the cache unconditionally. A build now **never narrows the cache**, guarded at the single save site rather than at `check`, because `show structure`, `describe`, `show catalog tables` and plain `refresh catalog` all reach it through the same seam and reproduce the bug identically. + + Two clauses go with the guard, and without them it trades a data-loss bug for a performance one. A rebuild that pays the full-mode cost anyway (`search`, `show references`, lint's graph rules) rebuilds at the level the project is set up for, so the cache comes back **current** instead of being refused and rebuilt on every invocation; and `REFRESH CATALOG` refreshes at the cached level, since "refresh" means bring what this project has up to date and there is no syntax for lowering the level. The cache's mode is therefore sticky — delete `.mxcli/catalog.db` to drop back to a cheaper one. + + A **fast** consumer is deliberately not upgraded: it is the hot path, the cache goes stale on every project save, and upgrading would put a source reindex inside the `exec` → `check` → `exec` loop. It builds narrow in memory instead, measured at +0.12s per invocation on the fixture (0.48s from cache, 0.60s rebuilt) — in exchange for a cache that no longer thrashes between three modes with the last builder destroying the others' tables. + - **An enumeration `decision` passed `check` and `exec`, then failed the build with CE6686** — "The current outcomes of the decision activity do not match the configured expression. Regenerate the outcomes." Mendix generates a decision's outcomes as one per enumeration value **plus one for the empty value**, and mxbuild compares the stored set against that generated set — so an enumeration decision written without `'' -> { }` is a build error mxcli had nothing to say about. `mxcli check` now reports it as **MDL-WF06** (error), which also refuses it at `exec` before anything is written. The severity turned on a measurement, not on the reading that suggests itself. "The empty branch must be for attributes that can be empty" is wrong: on mxbuild 11.10.0 a decision on an attribute carrying a **required (`not null`)** validation rule is still CE6686 without it. The condition is on the enumeration type, not on the value, so this is an error rather than a warning. diff --git a/mdl/executor/catalog_cache_mode_test.go b/mdl/executor/catalog_cache_mode_test.go new file mode 100644 index 0000000000..334a1a0c5e --- /dev/null +++ b/mdl/executor/catalog_cache_mode_test.go @@ -0,0 +1,220 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/mendixlabs/mxcli/mdl/catalog" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// The catalog cache records the mode it was built in, and the modes nest: +// source ⊃ full ⊃ fast. A consumer that only needs fast-mode answers — `mxcli +// check`, `show structure`, `describe` — must not be able to replace a richer +// cache with its own narrow one. That is mendixlabs/mxcli#1081: after the +// project file changes, a `check` rebuilt fast and saved over a source-mode +// cache, emptying source, refs, permissions, strings and xpath_expressions in +// one go. The commands that read those tables then report "requires refresh +// catalog full source" instead of an answer. +// +// A real project rather than a mock, for the same reason typecheck_test gives: +// the thing under test is a file on disk written from a model on disk. + +// sourceCachedFixture connects an executor to a copy of the shared fixture and +// builds a source-mode catalog, returning the executor and the cache path. +func sourceCachedFixture(t *testing.T) (*Executor, string) { + t.Helper() + exec := typeCheckFixture(t) + run(t, exec, "REFRESH CATALOG FULL SOURCE FORCE") + + ctx := exec.newExecContext(context.Background()) + cachePath := getCachePath(ctx) + if cachePath == "" { + t.Fatal("no cache path — the fixture is not connected to a project on disk") + } + if mode := cachedMode(t, cachePath); mode != "source" { + t.Fatalf("fixture setup: cache mode = %q, want %q", mode, "source") + } + if n := cachedRowCount(t, cachePath, "source"); n == 0 { + t.Fatal("fixture setup: source table is empty, so its loss would prove nothing") + } + return exec, cachePath +} + +// cachedMode reports the build mode recorded in the cache file. +func cachedMode(t *testing.T, cachePath string) string { + t.Helper() + cat, err := catalog.NewFromFile(cachePath) + if err != nil { + t.Fatalf("open cache %s: %v", cachePath, err) + } + defer cat.Close() + info, err := cat.GetCacheInfo() + if err != nil { + t.Fatalf("read cache info: %v", err) + } + return info.BuildMode +} + +// cachedRowCount counts the rows of one table inside the cache file. +func cachedRowCount(t *testing.T, cachePath, table string) int { + t.Helper() + cat, err := catalog.NewFromFile(cachePath) + if err != nil { + t.Fatalf("open cache %s: %v", cachePath, err) + } + defer cat.Close() + res, err := cat.Query("select count(*) from " + table) + if err != nil { + t.Fatalf("count %s: %v", table, err) + } + if len(res.Rows) == 0 { + return 0 + } + var n int + fmt.Sscanf(fmt.Sprintf("%v", res.Rows[0][0]), "%d", &n) + return n +} + +// touchProject moves the project file's mtime forward, which is what makes the +// cache invalid — the everyday trigger being a save in Studio Pro, an `mxcli +// exec`, or a branch switch between two mxcli commands. +func touchProject(t *testing.T, exec *Executor) { + t.Helper() + ctx := exec.newExecContext(context.Background()) + future := time.Now().Add(2 * time.Second) + if err := os.Chtimes(ctx.MprPath, future, future); err != nil { + t.Fatalf("touch project: %v", err) + } +} + +// TestCheckDoesNotDowngradeASourceCache is #1081 itself, entered through the +// tier `mxcli check -p` actually reaches. TypeCheckProgram asks for a fast +// catalog; with the cache stale it rebuilds, and the rebuild must not become +// the new cache. +func TestCheckDoesNotDowngradeASourceCache(t *testing.T) { + exec, cachePath := sourceCachedFixture(t) + before := cachedRowCount(t, cachePath, "source") + + touchProject(t, exec) + + prog, errs := visitor.Build(`CREATE MICROFLOW MyFirstModule.Probe() RETURNS BOOLEAN BEGIN RETURN true; END`) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + exec.TypeCheckProgram(prog) + + if mode := cachedMode(t, cachePath); mode != "source" { + t.Errorf("after check, cache mode = %q, want %q — a fast rebuild overwrote the source cache", mode, "source") + } + if n := cachedRowCount(t, cachePath, "source"); n != before { + t.Errorf("after check, source rows = %d, want %d — the source index was destroyed", n, before) + } +} + +// TestFastConsumersDoNotDowngradeASourceCache covers the rest of the fast-mode +// callers through their shared seam. `check` is where #1081 was reported, but +// `show structure`, `describe` and `show catalog tables` all call ensureCatalog +// the same way, so fixing this at the check command would have left the bug in +// place for them. +func TestFastConsumersDoNotDowngradeASourceCache(t *testing.T) { + exec, cachePath := sourceCachedFixture(t) + before := cachedRowCount(t, cachePath, "source") + + touchProject(t, exec) + + ctx := exec.newExecContext(context.Background()) + if err := ensureCatalog(ctx, false); err != nil { + t.Fatalf("ensureCatalog(fast): %v", err) + } + + if mode := cachedMode(t, cachePath); mode != "source" { + t.Errorf("after a fast rebuild, cache mode = %q, want %q", mode, "source") + } + if n := cachedRowCount(t, cachePath, "source"); n != before { + t.Errorf("after a fast rebuild, source rows = %d, want %d", n, before) + } +} + +// TestFullRebuildRestoresTheCachedSourceMode is the other half, and the reason +// "leave the cache alone" is not the whole fix. A consumer that needs full mode +// (search, show references, lint's graph rules) has to rebuild anyway; it +// rebuilds at the level the project is set up for, so the cache comes back +// current at source rather than thrashing on every invocation. +func TestFullRebuildRestoresTheCachedSourceMode(t *testing.T) { + exec, cachePath := sourceCachedFixture(t) + before := cachedRowCount(t, cachePath, "source") + + touchProject(t, exec) + + ctx := exec.newExecContext(context.Background()) + if err := ensureCatalog(ctx, true); err != nil { + t.Fatalf("ensureCatalog(full): %v", err) + } + + if mode := cachedMode(t, cachePath); mode != "source" { + t.Errorf("after a full rebuild, cache mode = %q, want %q — the source level was not restored", mode, "source") + } + if n := cachedRowCount(t, cachePath, "source"); n != before { + t.Errorf("after a full rebuild, source rows = %d, want %d", n, before) + } + // And the cache must be current again, or the next consumer rebuilds too. + if valid, reason := isCacheValid(ctx, cachePath, "source"); !valid { + t.Errorf("cache still invalid after a full rebuild: %s", reason) + } +} + +// TestPlainRefreshCatalogKeepsTheSourceMode covers the explicit statement. +// REFRESH CATALOG means "bring what this project has up to date", not "change +// its level" — there is no syntax for lowering the level, so treating a bare +// REFRESH CATALOG as a request to drop the source index is a silent loss. +func TestPlainRefreshCatalogKeepsTheSourceMode(t *testing.T) { + exec, cachePath := sourceCachedFixture(t) + before := cachedRowCount(t, cachePath, "source") + + touchProject(t, exec) + run(t, exec, "REFRESH CATALOG") + + if mode := cachedMode(t, cachePath); mode != "source" { + t.Errorf("after REFRESH CATALOG, cache mode = %q, want %q", mode, "source") + } + if n := cachedRowCount(t, cachePath, "source"); n != before { + t.Errorf("after REFRESH CATALOG, source rows = %d, want %d", n, before) + } + // And it has to leave a *current* cache. The never-narrow guard alone would + // keep the level by declining to save at all, which turns an explicit + // refresh into a no-op on disk — the cache stays source-mode and stale. + ctx := exec.newExecContext(context.Background()) + if valid, reason := isCacheValid(ctx, cachePath, "source"); !valid { + t.Errorf("cache still invalid after REFRESH CATALOG: %s", reason) + } +} + +// TestFirstBuildStillWritesItsOwnMode is the control on the guard: with no +// cache to protect, a fast build must still be cached as fast. A guard that +// simply stopped persisting fast catalogs would pass every test above and make +// every fast consumer rebuild from scratch forever. +func TestFirstBuildStillWritesItsOwnMode(t *testing.T) { + exec := typeCheckFixture(t) + ctx := exec.newExecContext(context.Background()) + cachePath := getCachePath(ctx) + if err := os.RemoveAll(filepath.Dir(cachePath)); err != nil { + t.Fatalf("clear cache dir: %v", err) + } + + if err := ensureCatalog(ctx, false); err != nil { + t.Fatalf("ensureCatalog(fast): %v", err) + } + if _, err := os.Stat(cachePath); err != nil { + t.Fatalf("no cache written on a first fast build: %v", err) + } + if mode := cachedMode(t, cachePath); mode != "fast" { + t.Errorf("first build cache mode = %q, want %q", mode, "fast") + } +} diff --git a/mdl/executor/cmd_catalog.go b/mdl/executor/cmd_catalog.go index 842136f98d..3fae05572f 100644 --- a/mdl/executor/cmd_catalog.go +++ b/mdl/executor/cmd_catalog.go @@ -220,6 +220,52 @@ func execDescribeCatalogTable(ctx *ExecContext, stmt *ast.DescribeCatalogTableSt return writeResult(ctx, tr) } +// Catalog build modes nest: source ⊃ full ⊃ fast. A source catalog answers +// everything a full one does and adds the MDL source index; a full one answers +// everything a fast one does and adds activities, widgets, refs, permissions, +// strings and xpath. catalogModeRank orders them so the two questions that +// matter — "does this cache satisfy what I need?" and "would writing this cache +// lose something?" — are both comparisons rather than case analysis. +// +// An unknown mode ranks 0, below every real one, so it never masquerades as +// something richer than it is. +func catalogModeRank(mode string) int { + switch mode { + case "fast": + return 1 + case "full": + return 2 + case "source": + return 3 + } + return 0 +} + +// cachedCatalogMode reports the build mode recorded in the on-disk cache, or "" +// when there is no readable cache. Deliberately indifferent to whether that +// cache is still *valid*: a stale source-mode cache is stale data but a live +// statement of the level this project is set up for, and that is what callers +// here are asking about. +func cachedCatalogMode(ctx *ExecContext) string { + cachePath := getCachePath(ctx) + if cachePath == "" { + return "" + } + if _, err := os.Stat(cachePath); err != nil { + return "" + } + cat, err := catalog.NewFromFile(cachePath) + if err != nil { + return "" + } + defer cat.Close() + info, err := cat.GetCacheInfo() + if err != nil { + return "" + } + return info.BuildMode +} + // ensureCatalog ensures a catalog is available, using cache if possible. func ensureCatalog(ctx *ExecContext, full bool) error { requiredMode := "fast" @@ -241,8 +287,24 @@ func ensureCatalog(ctx *ExecContext, full bool) error { return mdlerrors.NewNotConnected() } + // A rebuild that is going to pay the full-mode cost anyway rebuilds at the + // level the project is set up for, so the cache comes back current instead + // of being refused by buildCatalog's never-narrow guard on every call. + // Without this, `search` / `show references` on a source-mode project would + // rebuild full, decline to save, and do it again next time. + // + // A *fast* consumer is deliberately not upgraded. It is the hot path — the + // tier `mxcli check -p`, `show structure` and `describe` reach on every + // invocation, and the cache goes stale on every project save — so making it + // pay for a source reindex would trade one papercut for a worse one. It + // builds narrow in memory and the guard keeps its result off disk. + isSource := false + if full && cachedCatalogMode(ctx) == "source" { + isSource = true + } + // Build fresh catalog - return buildCatalog(ctx, full, false, false, 0) + return buildCatalog(ctx, full, isSource, false, 0) } // getCachePath returns the path to the catalog cache file for the current project. @@ -298,9 +360,8 @@ func isCacheValid(ctx *ExecContext, cachePath string, requiredMode string) (bool } // Check build mode hierarchy: source > full > fast - modeRank := map[string]int{"fast": 1, "full": 2, "source": 3} - cachedRank := modeRank[info.BuildMode] - requiredRank := modeRank[requiredMode] + cachedRank := catalogModeRank(info.BuildMode) + requiredRank := catalogModeRank(requiredMode) if requiredRank > cachedRank { return false, fmt.Sprintf("%s mode requested but cache is %s mode", requiredMode, info.BuildMode) } @@ -416,9 +477,28 @@ func buildCatalog(ctx *ExecContext, full, isSource, communities bool, resolution fmt.Fprintf(ctx.Output, "✓ Catalog ready (%.1fs)\n", elapsed.Seconds()) } - // Save to cache file + // Save to cache file — unless doing so would narrow it. + // + // The cache records the mode it was built in, and a build that needs less + // than the cache holds must not become the new cache. mendixlabs/mxcli#1081: + // once the project file changed, a `mxcli check -p` rebuilt fast (all it + // needs is attribute types and enum cases) and saved over a source-mode + // cache, taking source, refs, permissions, strings and xpath_expressions + // with it. The commands that read those tables then answered "requires + // refresh catalog full source" instead of answering. + // + // This is the choke point on purpose. `refresh catalog communities` was + // given a local fix for the identical hazard — its comment below still + // describes it — and every other caller kept the bug, which is exactly the + // shape of failure a guard at one call site produces. cachePath := getCachePath(ctx) if cachePath != "" { + if existing := cachedCatalogMode(ctx); catalogModeRank(existing) > catalogModeRank(buildMode) { + if !ctx.Quiet { + fmt.Fprintf(ctx.Output, "Keeping the existing %s-mode catalog cache (this build was %s mode)\n", existing, buildMode) + } + return nil + } cacheDir := filepath.Dir(cachePath) if err := os.MkdirAll(cacheDir, 0755); err == nil { // Remove existing cache file first @@ -507,6 +587,25 @@ func execRefreshCatalogStmt(ctx *ExecContext, stmt *ast.RefreshCatalogStmt) erro return nil } + // REFRESH CATALOG means "bring what this project has up to date", not + // "change its level". There is no syntax for lowering the level, so reading + // a bare REFRESH CATALOG as a request to drop the source index would make + // the loss both silent and unaskable-for. Refresh at least what is cached. + full, source := stmt.Full, stmt.Source + switch cachedCatalogMode(ctx) { + case "source": + full, source = true, true + case "full": + full = true + } + if (full != stmt.Full || source != stmt.Source) && !ctx.Quiet { + mode := "full" + if source { + mode = "source" + } + fmt.Fprintf(ctx.Output, "Refreshing at %s mode to match the existing cache\n", mode) + } + // Close existing catalog if any if ctx.Catalog != nil { ctx.Catalog.Close() @@ -522,7 +621,7 @@ func execRefreshCatalogStmt(ctx *ExecContext, stmt *ast.RefreshCatalogStmt) erro bgCtx.Output = sw // background goroutine writes through sw syncCatalog := ctx.SyncCatalog // capture callback before returning go func() { - if err := buildCatalog(&bgCtx, stmt.Full, stmt.Source, stmt.Communities, stmt.Resolution); err != nil { + if err := buildCatalog(&bgCtx, full, source, stmt.Communities, stmt.Resolution); err != nil { fmt.Fprintf(bgCtx.Output, "Background catalog build failed: %v\n", err) return } @@ -561,7 +660,7 @@ func execRefreshCatalogStmt(ctx *ExecContext, stmt *ast.RefreshCatalogStmt) erro } // Rebuild the catalog - return buildCatalog(ctx, stmt.Full, stmt.Source, stmt.Communities, stmt.Resolution) + return buildCatalog(ctx, full, source, stmt.Communities, stmt.Resolution) } // execRefreshCatalog handles REFRESH CATALOG [FULL] command (legacy signature). diff --git a/mdl/executor/cmd_misc.go b/mdl/executor/cmd_misc.go index ffab2109f4..8c2118667f 100644 --- a/mdl/executor/cmd_misc.go +++ b/mdl/executor/cmd_misc.go @@ -214,7 +214,10 @@ Catalog Queries: (* only populated with refresh catalog full) (** only populated with refresh catalog full source) - Cache is stored in .mxcli/catalog.db next to the .mpr file. + Cache is stored in .mxcli/catalog.db next to the .mpr file. Its mode is + sticky: once a project has a full or source cache, a refresh rebuilds at + that level and a command needing less never writes a narrower one over it. + To drop back to a cheaper level, delete .mxcli/catalog.db and refresh. Code Search (requires refresh catalog full): show callers of Module.Microflow [transitive]; From afa4742ada75c9cbd16a91fe96079370b14a52b4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 14:46:05 +0000 Subject: [PATCH 19/19] fix(microflow): keep an annotation's identity through a round trip (mendixlabs/mxcli#1077) A note connected to several activities came back DUPLICATED after describe -> exec: one Microflows$Annotation with N AnnotationFlows became N notes with one flow each. Measured on Mendix 11.13.0 with mxbuild at 0 errors on both sides, which is why it had never been caught -- a duplicated note is a perfectly valid model. The reporter's closing line is the diagnosis: "MDL currently does not (really) support one annotation linked to multiple activities." In Mendix a note is a NODE with edges; MDL modelled it as one string per activity. A per-activity string cannot express identity, so buildAnnotationsByTarget discarded it at the join and attachAnnotation minted a fresh Annotation per mention. The relation is many-to-many and the string lost BOTH directions, so the duplication report was hiding a deletion: A. one note -> N activities came back as N notes (reported) B. N notes -> one activity came back as ONE (found here) B is worse: the AST had a single AnnotationText slot and the last line won, so the others were gone from the model. A third loss rode along -- an Annotation's own position and size were never emitted and the writer re-invented them, moving and resizing every note on every round trip. Notes now have identity in the language: @annotation(id: n1, text: 'shared', position: (x, y), size: (w, h)) @annotation(id: n1) -- attaches THAT note to another activity `@annotation 'text'` is unchanged, is still repeatable, and is still what DESCRIBE emits: position/size are omitted whenever they match what the writer re-derives (both sides go through defaultAnnotationGeometry), and `id:` is emitted only for a note that really is shared. Two notes with identical text and no id stay two notes -- mxcli never merges on content. The grammar needed `text` and `position` in annotationParamName. A keyword parameter key does not FAIL to parse -- annotationParam falls through to its positional alternative -- so those forms were being accepted and silently ignored. MDL079 now refuses a parameter the visitor cannot use, and a reference to a note that has no text; it walks in statement order so `check` and `exec` refuse the same scripts. Three adjacent defects, each with a failing test first: - a note inside `on error { ... }` arrived DETACHED (the handler sub-builder's merge copied objects and sequence flows but not annotation flows) and was never described at all (collectErrorHandlerStatements is a second describer) - refusals raised while building a loop body were swallowed, so exec reported success on a flow it had written wrong - StatementAnnotations named a coverage test that was never written; written now, alongside one for the new StatementBodies walker Verified on Mendix 11.13.0 / mxbuild 11.13, both engines byte-identical: shared note round-trips to 1 note + 2 flows reporting "Unchanged microflow"; two stacked notes both survive and do not overlap; geometry (175,-40) 260x70 preserved; describe -> exec -> describe byte-identical on all five bug-test microflows; mx check 0 errors throughout. Controls: stubbing the identity flag reproduces "got 2 notes, want 1"; restoring the single AST slot reproduces "got [second note], want both". Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 3 + .../skills/mendix/write-microflows/SKILL.md | 23 + cmd/mxcli/syntax/features_microflow.go | 16 +- docs/01-project/MDL_QUICK_REFERENCE.md | 6 +- .../microflow-1077-annotation-sharing.mdl | 130 +++++ mdl/ast/annotations.go | 57 ++ mdl/ast/annotations_coverage_test.go | 204 +++++++ mdl/ast/ast_microflow.go | 63 +- mdl/executor/cmd_microflows_builder.go | 25 +- .../cmd_microflows_builder_annotations.go | 143 +++-- ...cmd_microflows_builder_annotations_test.go | 36 +- .../cmd_microflows_builder_control.go | 40 ++ mdl/executor/cmd_microflows_builder_flows.go | 17 +- mdl/executor/cmd_microflows_builder_graph.go | 15 +- ...icroflows_describe_loop_annotation_test.go | 9 +- mdl/executor/cmd_microflows_show_helpers.go | 258 +++++++-- .../cmd_microflows_show_helpers_test.go | 20 +- mdl/executor/cmd_microflows_traverse_test.go | 19 +- .../microflow_annotation_sharing_test.go | 542 ++++++++++++++++++ mdl/executor/validate_microflow.go | 50 ++ .../validate_microflow_loop_caption_test.go | 2 +- mdl/grammar/domains/MDLSettings.g4 | 8 + mdl/visitor/visitor_microflow_statements.go | 106 +++- mdl/visitor/visitor_test.go | 21 +- 24 files changed, 1641 insertions(+), 172 deletions(-) create mode 100644 mdl-examples/bug-tests/microflow-1077-annotation-sharing.mdl create mode 100644 mdl/ast/annotations_coverage_test.go create mode 100644 mdl/executor/microflow_annotation_sharing_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index efd2092ffc..b34da5360c 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -576,3 +576,6 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A microflow whose activity has a CUSTOM error handler comes back from `describe microflow` with the handler AND every activity in its branch missing. Output stays valid MDL, `mxcli check` passes, nothing on stderr \u2014 so describe\u2192edit\u2192exec deletes the handler from the model. Reported on a create-variable activity with \"custom with rollback\" (v0.21, Mx 11.12.3); reproduced on HEAD/11.14.0 with a pure mxcli round trip", "cause": "TWO STACKED DEFECTS. (1) `getActionErrorHandlingType` was a hand-maintained switch covering 17 of the 38 action types that store ErrorHandlingType. `emitActivityStatement` walks the error branch only when `hasCustomErrorHandler(errType)` agrees, so each of the 21 missing types lost the ENTIRE `on error { \u2026 }` block, not just the suffix \u2014 CreateObject and ChangeObject among them. (2) legacy only: 9 parse functions never read ErrorHandlingType off the BSON, so the value was gone before the describer was asked. Fixing (1) alone left legacy still broken, which is how they hid each other", "file": "`mdl/executor/cmd_microflows_show_helpers.go` (`getActionErrorHandlingType` \u2192 new `actionErrorHandlingField`); `sdk/mpr/parser_microflow.go`, `sdk/mpr/parser_microflow_actions.go` (9 parsers); grammar+visitor+builder for 8 statements; `mdl/executor/validate_microflow_error_handling.go` (MDL076 table, new MDL077)", "insight": "**Count the gap before fixing the instance** \u2014 the reported activity was 1 of 21, measured by diffing the switch's cases against the action types declaring the field. **Replace the enumeration, do not extend it**: the list had already been patched per-instance (#863) and silently regrew, so it is now a reflection lookup on the `ErrorHandlingType` field, with `RestOperationCallAction` the ONE deliberate exclusion (Mendix rejects a custom handler there, CE6035). Actions embed model.BaseElement, which has no such field, so no promoted field is picked up by accident. **The trap: fixing the describer alone makes things WORSE.** 8 statement forms had no `onErrorClause` in the grammar \u2014 `declare` (the reporter's own), `set`, `change`, `log`, `show page`, `close page`, `show message`, `validation feedback` \u2014 so DESCRIBE began emitting `declare $name String = 'v' on error { \u2026 };`, which fails to parse (`mismatched input 'on' expecting ';'`). Trading a silent drop for a broken script is not a fix; the grammar was extended instead. **Measured on 11.14.0, and the result is not guessable**: all 8 accept a custom handler (0 errors), but for `on error continue` create-VARIABLE and change-VARIABLE are fine while change-OBJECT, log, show page, close page, show message and validation feedback are CE6035 \u2014 now in MDL076's deny-list, whose comment claiming Log/Change were \"unreachable from a script\" this change invalidated. New MDL077 refuses `on error` on the list-operation/aggregate forms of `set`, which genuinely have no ErrorHandlingType in the metamodel \u2014 one MDL keyword spanning activities that can and cannot hold the clause. **A non-terminating handler causing CE0108 is NOT a bug**: the branch merges back and a later variable is out of scope on the error path; Studio Pro reports the same. Controls: revert the lookup \u2192 all 7 describer cases fail naming the dropped branch; revert one parser \u2192 the legacy case fails; a no-clause microflow must still render NO suffix (#840 in reverse). Repro `mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl`; describe\u2192exec\u2192describe byte-identical and reported \"Unchanged microflow\"", "refs": ["#1078", "#863", "#840"], "ce": ["CE6035", "CE0108"]} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "CI `make test-integration` fails where `go test ./...` is green: `TestMxCheck_DoctypeScripts/02b-nanoflow-examples.mdl` (both engines) and `/03-page-examples.mdl` report **CE6035 \"Error handling type is not supported\"** on 11 activities \u2014 every un-annotated Change object / Log message / Validation feedback / Close page in a NANOFLOW. Self-inflicted while fixing #1078", "cause": "Eight builders were switched from `fb.ehType(nil)` to `explicitErrorHandling(fb, s.ErrorHandling)` on the action. `explicitErrorHandling` returns EMPTY for \"no clause\", and the writers turn empty into a literal `\"Rollback\"` via orDefault \u2014 but `fb.ehType(nil)` is CONTEXT-DEPENDENT and returns **Abort** in a nanoflow. So every un-annotated nanoflow activity silently changed from Abort to Rollback, which mxbuild rejects", "file": "`mdl/executor/cmd_microflows_builder_actions.go` + `cmd_microflows_builder_calls.go` (8 sites reverted to `fb.ehType(s.ErrorHandling)`); `mdl/executor/nanoflow_validation.go` (`getErrorHandling`)", "insight": "**The same helper is correct for one action and wrong for its neighbour, and the difference is what the code did BEFORE.** `explicitErrorHandling` is right for Retrieve/Delete (#1020-era): their writers emitted a hardcoded `\"Rollback\"` that those two actions accept in every flow flavour, so empty\u2192Rollback is a no-op. It is wrong wherever the builder already supplied a *context-dependent* default \u2014 empty discards the flow flavour. The rule: before replacing a default-supplying expression, ask what the OLD expression returned in every context, not just the one you are testing. Its own doc comment names Abort/nanoflow and I still missed it, because I was reading it as \"the safe choice\" rather than \"the choice that preserves THIS call site's prior value\". **`go test ./...` does not run this suite** \u2014 the mx-check round trips are behind `-tags integration` (`make test-integration`, ~30 min), so a green unit suite says nothing about whether mxbuild still accepts what mxcli writes; run it before pushing anything that touches a serialized default. Regression test `TestAuthorOnError_NanoflowKeepsAbortWithoutAClause` (control: the same statements in a microflow must NOT become Abort) \u2014 a unit test, so it catches this in seconds instead of 30 minutes. A second, quieter gap in the same change: `getErrorHandling` in nanoflow_validation.go gates the walk that looks for disallowed actions INSIDE a handler body, so the eight new statements had to be added there too or a Java action nested in `declare \u2026 on error { \u2026 }` goes unreported (test carries a `commit` control, since that entry point is `validateNanoflowBody`, NOT the exported `ValidateNanoflowBody`, which is a different check)", "refs": ["#1078"], "ce": ["CE6035"]} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "In a NANOFLOW, `log ... on error continue|rollback|{ \u2026 }`, and a custom handler on change/show page/close page/show message/validation feedback, are written by mxcli and rejected by mxbuild as **CE6035 \"Error handling type is not supported\"**. `mxcli check` passed. Reachable for the first time via mendixlabs/mxcli#1078, which gave those statements an onErrorClause", "cause": "No rule covered it. MDL076 runs on the microflow validator, which has no flow flavour, so it cannot express \"CE6035 in a nanoflow but fine in a microflow\" \u2014 and every one of these six IS fine in a microflow with a custom handler", "file": "`mdl/executor/nanoflow_validation.go` (`checkNanoflowErrorHandling`, `nanoflowErrorHandlingUnsupported`, wired into `validateNanoflowStatements`)", "insight": "**A nanoflow accepts error handling on almost nothing**: measured on 11.14.0, only the two VARIABLE activities (create-variable, change-variable) take a clause; the other six are CE6035 whichever form is written, because a nanoflow activity's only accepted value is Abort \u2014 the no-clause default, which no MDL syntax writes. So the rule refuses the CLAUSE, not one spelling. The split is by ACTIVITY, not by client-side/server-side: `show message` is as client-side as it gets and still refuses one, while `declare` accepts it. Same permissive pair as `continue` in a microflow, which is the only pattern visible across both tables. **Two validators, two entry points, easy to wire to the wrong one**: `validateNanoflowBody` (unexported) is the disallowed-action walk that exec runs; `ValidateNanoflowBody` (exported) is the variable/semantic check that `check` runs. A test targeting the exported one passes vacuously \u2014 caught only because the test carried a `commit` control that also failed. Note the pre-existing consequence: every nanoflow restriction, including the 22 disallowed action types, is enforced at EXEC and not by `mxcli check` (even with `--references`) \u2014 a check/exec divergence worth closing on its own, but not in a bug fix, since it would newly reject scripts that pass check today", "refs": ["#1078"], "ce": ["CE6035"]} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "A microflow note (Microflows$Annotation) connected to several activities came back DUPLICATED after describe -> exec: one note with N AnnotationFlows became N notes with one flow each. Measured on Mendix 11.13.0: 1 note + 2 flows -> 2 notes. mx check: 0 errors on both sides, so nothing warned.", "cause": "buildAnnotationsByTarget joined AnnotationFlows to Annotation objects and kept only the CAPTION, filing it under each flow's destination. The note's identity was discarded at that join, so the describer had no way to know two lines were the same note, and MDL had no way to say so either: ActivityAnnotations carried AnnotationText as a single STRING per activity. attachAnnotation then minted a fresh Annotation + flow per line.", "file": "mdl/executor/cmd_microflows_show_helpers.go (buildAnnotationsByTarget, annotationEmitter), mdl/executor/cmd_microflows_builder_annotations.go (attachAnnotation, defaultAnnotationGeometry), mdl/ast/ast_microflow.go (MicroflowAnnotation), mdl/visitor/visitor_microflow_statements.go (parseNoteAnnotation), mdl/grammar/domains/MDLSettings.g4 (annotationParamName)", "insight": "The relation is MANY-TO-MANY and a per-activity string lost BOTH directions — which is how a duplication report turned out to be hiding a deletion. One note to N activities duplicated (the report); N notes to one activity kept only the LAST, because the visitor assigned into the single slot. The second is strictly worse and nobody had noticed it. When a report says a round trip 'duplicates' something, check the transpose before fixing: the same missing cardinality usually destroys in the other direction. The fix that holds is giving the construct IDENTITY in the language (`id:`) rather than deduplicating on content — merging two notes because their text matches would be a second silent rewrite, so the control test is two identical-text notes that must STAY two.", "refs": ["mendixlabs/mxcli#1077"], "ce": []} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "A parameterised annotation whose key is a lexer keyword parses and is SILENTLY IGNORED. `@annotation(text: 'x')` and `@annotation(position: (1, 2))` were accepted by the grammar, reached no visitor case, and vanished; `@annotation(size: (1, 2))` and `@annotation(zz: (1, 2))` worked.", "cause": "annotationParam is `annotationParamName COLON value | annotationValue`, and annotationParamName lists only IDENTIFIER plus a hand-maintained set of keywords (FROM, TO, TRUE, FALSE, TAIL). A keyword key does not FAIL the parse — it falls through to the positional alternative — so the parameter is consumed and means nothing.", "file": "mdl/grammar/domains/MDLSettings.g4 (annotationParamName), mdl/visitor/visitor_microflow_statements.go", "insight": "The failure mode of that grammar rule is silence, not a parse error, so a new annotation parameter cannot be assumed to work because a probe script parsed clean — probe that the VALUE arrives, not that the text is accepted. Words that are already MDL keywords are exactly the readable ones you reach for (`text`, `position`, `size`, `caption`), so this will keep recurring; anything added must be listed in annotationParamName. The visitor now records an unusable parameter on ActivityAnnotations.InvalidNotes and MDL079 refuses it, matching how @curve's InvalidCurves feeds MDL060.", "refs": ["mendixlabs/mxcli#1077", "mendixlabs/mxcli#884"], "ce": []} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "A note written inside an `on error { … }` body arrived DETACHED — the Annotation object reached the model, its AnnotationFlow did not, so the note floated free on the canvas. DESCRIBE then never emitted it at all. Separately, any refusal raised while building a loop body was swallowed and exec reported success.", "cause": "Both sub-builders collect into their own slices and the merge back into the parent was incomplete: the error-handler merge copied objects and sequence flows but not annotationFlows or errors, and the loop merge copied annotationFlows but not errors. On the read side, collectErrorHandlerStatements is a SECOND describer (the main traversal never steps inside a handler block) and emitted no annotations.", "file": "mdl/executor/cmd_microflows_builder_flows.go (addErrorHandlerFlow merge), mdl/executor/cmd_microflows_builder_control.go (loop sub-builders), mdl/executor/cmd_microflows_show_helpers.go (collectErrorHandlerStatements)", "insight": "A sub-builder is a second copy of the builder's state and every field it collects needs an explicit line in the merge — the ones that get forgotten are the rare ones (annotationFlows) and the ones whose absence looks like success (errors). Grep the sub-builder's struct fields against the merge block rather than trusting it. The read side has the same shape: collectErrorHandlerStatements duplicates the traversal, so anything added to the main describer has to be added there too or the round trip loses it one nesting level down — fixing only the write half would have left the note attached in the model and still absent from DESCRIBE, which is differently wrong rather than fixed.", "refs": ["mendixlabs/mxcli#1077"], "ce": []} diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 6574cf5ee0..3615d291b5 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -597,6 +597,29 @@ $var/Module.AssociationName/attribute -- Chained commit $Order; -- Annotations apply here ``` +### Annotations Are Notes, and a Note Can Be Shared + +A note is a node with edges in Mendix, not a property of the activity it +documents. So `@annotation` is **repeatable** — one activity can carry several, +each its own note — and one note can be attached to several activities: + +```mdl +@annotation(id: n1, text: 'both of these touch the same record') +commit $Order; +@annotation(id: n1) -- attaches THAT note, does not copy it +commit $Invoice; +``` + +`id:` is scoped to the flow you are writing and is not stored in the model; it +exists only so a second mention can point at the first. **Without it, two lines +with identical text are two separate notes** — mxcli never merges on text. + +A note's own canvas geometry is `position: (x, y)` and `size: (w, h)`, e.g. +`@annotation(text: 'note', position: (175, -40), size: (260, 70))`. Omit them +and the note goes 100px above the activity at 200×50, stacking 60px per extra +note; DESCRIBE omits them again whenever they match, so an ordinary note keeps +the short `@annotation 'text'` form. + ### Execute Database Query Pattern ```mdl -- Static query (3-part name: Module.Connection.Query) diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index a182490f5d..edef419076 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -303,7 +303,9 @@ func init() { "@anchor(from: right, to: left) -- which SIDE each end of the outgoing flow attaches to\n" + "@curve(from: (40, -90), to: (-40, 90)) -- the flow's bezier control vectors\n" + "@merge(x, y) -- the implicit merge that closes a split\n" + - "@caption 'text'\n@color Green\n@annotation 'a note'\n@excluded\n\n" + + "@caption 'text'\n@color Green\n@annotation 'a note'\n@excluded\n" + + "@annotation(id: n1, text: 'a note', position: (x, y), size: (w, h))\n" + + "@annotation(id: n1) -- attaches THAT note to another activity\n\n" + "An unrecognised @name is an error (MDL059): it would parse and do nothing,\n" + "so a typo of @position would silently discard the layout.\n\n" + "Mendix stores no waypoints — a flow's shape is two control vectors, each a\n" + @@ -322,7 +324,17 @@ func init() { "parameters form a row along the top of the canvas at 200;53, 300;53, … ;\n" + "the same derived/authored rule as @start then applies, so a parameter on\n" + "that row is re-derived and one anywhere else survives a rewrite and is\n" + - "emitted by DESCRIBE.", + "emitted by DESCRIBE.\n\n" + + "A NOTE is a node with edges, not a property of the activity it documents:\n" + + "one note can be wired to several activities and several notes to one. So\n" + + "@annotation is repeatable, and `id:` names a note so a later\n" + + "@annotation(id: …) attaches the same one instead of creating a copy. The id\n" + + "is scoped to the flow being authored and is not stored in the model —\n" + + "DESCRIBE re-derives labels, and emits one only for a note that really is\n" + + "shared. Two notes with identical text and no id stay two notes.\n\n" + + "position:/size: are the note's own geometry, omitted whenever they match\n" + + "what a rewrite re-derives (100px above the activity, stacked 60px per extra\n" + + "note, 200x50), so an ordinary note keeps the short form. (#1077)", Example: "create microflow MyModule.ACT_Flow (\n @position(145, 0)\n $In: String\n)\nreturns String as $Out\nbegin\n" + " @start(145, 100)\n @position(200, 100)\n @anchor(from: bottom, to: top)\n" + " @curve(from: (40, -90), to: (-40, 90))\n declare $Tmp String = $In;\n" + diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index e4037ebb5b..e7ac490ac4 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -540,8 +540,10 @@ it is for pages. | Start event | `@start(x, y)` | Canvas position of the start, on the **first** statement. Omit it and the start is placed one spacing unit left of the first activity and MOVES with it on a rewrite; a start that is not at that derived spot is treated as hand-placed, survives a rewrite, and is emitted by DESCRIBE (#951) | | Caption | `@caption 'text'` | Custom caption (before activity) | | Color | `@color Green` | Background color (before activity) | -| Annotation | `@annotation 'text'` | Visual note attached to next activity | -| Free annotation | `@annotation 'text'` before `@position(...)` | Free-floating visual note preserved by order | +| Annotation | `@annotation 'text'` | Visual note attached to next activity. **Repeatable** — an activity can carry several, and each is its own note | +| Shared annotation | `@annotation(id: n1, text: 'note')` then `@annotation(id: n1)` | ONE note wired to several activities, which is how Mendix stores it. Without the `id:` the two lines are two separate notes, even with identical text. The id is scoped to the flow being authored and is not stored (#1077) | +| Annotation geometry | `@annotation(text: 'note', position: (x, y), size: (w, h))` | The note's own place and box on the canvas. Both are omitted whenever they match what a rewrite re-derives — 100px above the activity, stacked 60px per extra note, at 200×50 — so an ordinary note stays on the short form | +| Free annotation | `@annotation 'text'` before `@position(...)` | Free-floating visual note preserved by order. A free note has no activity to be placed relative to, so DESCRIBE always emits its `position:` | | IF | `if condition then ... [else ...] end if;` | | | Enum split | `case $Var when Value then ... end case;` | Enumeration decision branches. Bare enum values (never quoted or qualified), one branch per value **including `(empty)`** (MDL056), no `else` (MDL008), no `AS` alias | | Type split | `split type $Var when Module.Entity then ... when (empty) then ... end split;` | Runtime specialization branches. Same `when ... then` shape as the enum split. Needs a branch per subtype **and** the base entity (CE0090); `when (empty) then` is the **null-object** flow, not a default, and cannot be omitted (CE0089). Legacy `case Module.Entity` / `else` still parse (MDL065 warns) | diff --git a/mdl-examples/bug-tests/microflow-1077-annotation-sharing.mdl b/mdl-examples/bug-tests/microflow-1077-annotation-sharing.mdl new file mode 100644 index 0000000000..aa3e02af8d --- /dev/null +++ b/mdl-examples/bug-tests/microflow-1077-annotation-sharing.mdl @@ -0,0 +1,130 @@ +-- ============================================================================ +-- mendixlabs/mxcli#1077: a note wired to several activities came back copied +-- ============================================================================ +-- +-- Report: "When I have an annotation linked to multiple activities, in the MDL +-- the @annotation is duplicated. When regenerating the microflow, the annotation +-- is created multiple times instead of one time. MDL currently does not (really) +-- support one annotation linked to multiple activities." +-- mxcli v0.21 / Mendix 11.12.3. +-- +-- The closing sentence is the diagnosis. In Mendix a note is a NODE with edges — +-- one Microflows$Annotation joined to any number of activities by +-- Microflows$AnnotationFlow — and MDL modelled it as one string per activity. +-- A per-activity string cannot express identity, so DESCRIBE filed the caption +-- once per target and re-executing built one note per mention. +-- +-- --------------------------------------------------------------------------- +-- THE SAME ROOT CAUSE, POINTING THE OTHER WAY, IS WORSE +-- --------------------------------------------------------------------------- +-- The relation is many-to-many, and the string lost BOTH directions: +-- +-- A. one note → N activities came back as N notes (the report) +-- B. N notes → one activity came back as ONE note (found here) +-- +-- B is not a duplication, it is a DELETION: the AST had a single AnnotationText +-- slot and the last @annotation line won, so the others were gone from the model +-- after one describe → exec. Measured on Mendix 11.13.0: two notes on an +-- activity, round trip, and 'note one' no longer exists. +-- +-- A third loss rode along: an Annotation's own position and size were never +-- emitted and the writer re-invented them, so every round trip moved every note +-- to (activity.x, activity.y-100) and resized it to 200x50 — measured, a note at +-- (175, -40) sized 260x70 came back at (360, 100) sized 200x50. +-- +-- `mx check` reported 0 errors on every side of all three. A duplicated, +-- deleted, or displaced note is a perfectly valid model, which is why none of +-- this had ever been caught: mxbuild is not an oracle for a describer. +-- +-- --------------------------------------------------------------------------- +-- WHAT CHANGED IN THE SYNTAX +-- --------------------------------------------------------------------------- +-- `@annotation 'text'` is unchanged and is still what you write and what +-- DESCRIBE emits for the ordinary case. A note only pays for the longer form +-- when it has something the short form cannot carry: +-- +-- @annotation(id: n1, text: 'shared', position: (x, y), size: (w, h)) +-- @annotation(id: n1) -- attaches THAT note to another activity +-- +-- `id:` is scoped to the flow being authored and is not stored in the model; +-- the describer re-derives labels, so they are stable across a round trip by +-- construction rather than by being remembered. `position:`/`size:` are omitted +-- whenever they match what the writer would re-derive, which is what keeps +-- existing scripts and existing DESCRIBE output unchanged. +-- +-- The grammar needed two words: `position` and `text` are lexer keywords, and a +-- keyword parameter key does NOT fail to parse — annotationParam falls through +-- to its positional alternative, so the parameter is accepted and silently means +-- nothing. `@annotation(text: 'x')` parsed and was discarded before this fix. +-- +-- --------------------------------------------------------------------------- +-- MEASURED on Mendix 11.13.0 / mxbuild 11.13 +-- --------------------------------------------------------------------------- +-- Each microflow below: exec, `describe microflow`, exec the output, describe +-- again. The two describes are byte-identical, and the shared case reports +-- "Unchanged microflow" on the re-exec — the rebuild was semantically equal to +-- what was stored, so nothing was even written. Both engines (default and +-- MXCLI_ENGINE=legacy) produce identical output; unlike #1078 this defect was +-- never engine-specific. +-- +-- Entities assumed: none. Adjust the log node name for your project. +-- ============================================================================ + +-- The reporter's shape: ONE note documenting two activities. Before the fix, +-- describing this and running the output back created a second note. +create or modify microflow MyFirstModule.ACT_1077_Shared () returns String +begin + @annotation(id: warn, text: 'both of these touch the same record', position: (175, -40), size: (260, 70)) + log info node 'Bug1077' 'first'; + @annotation(id: warn) + log info node 'Bug1077' 'second'; + return 'ok'; +end; + +-- Defect B: two DIFFERENT notes on one activity. Before the fix only the last +-- survived. Neither is positioned here, and neither has to be: unplaced notes +-- stack upwards (100px above the activity, then 60px per note) rather than +-- landing on top of each other. DESCRIBE gives both back on the short form, +-- because that stacking is exactly what it re-derives. +create or modify microflow MyFirstModule.ACT_1077_Stacked () returns String +begin + @annotation 'the first thing to know' + @annotation 'the second thing to know' + log info node 'Bug1077' 'busy step'; + return 'ok'; +end; + +-- A note whose text is the same on two activities but which are genuinely two +-- separate notes. This is the control for the shared case above: identical text +-- must NOT be merged, or the fix would be trading one silent rewrite for +-- another. +create or modify microflow MyFirstModule.ACT_1077_SameTextTwice () returns String +begin + @annotation 'check this' + log info node 'Bug1077' 'first'; + @annotation 'check this' + log info node 'Bug1077' 'second'; + return 'ok'; +end; + +-- A note inside an `on error` handler body. Its Annotation object reached the +-- model but its AnnotationFlow did not — the handler's sub-builder collected +-- annotation flows into its own slice and the merge back copied objects and +-- sequence flows only — so the note arrived detached, floating on the canvas. +create or modify microflow MyFirstModule.ACT_1077_InHandler () returns String +begin + declare $name String = 'value' on error { + @annotation 'this is where it goes wrong' + log error node 'Bug1077' 'boom'; + return 'failed'; + }; + return $name; +end; + +-- Control for the whole file: no notes at all, so DESCRIBE must emit none and +-- the round trip must not invent one. +create or modify microflow MyFirstModule.ACT_1077_NoNotes () returns String +begin + log info node 'Bug1077' 'nothing to see'; + return 'ok'; +end; diff --git a/mdl/ast/annotations.go b/mdl/ast/annotations.go index 2c807837be..891e8d8eae 100644 --- a/mdl/ast/annotations.go +++ b/mdl/ast/annotations.go @@ -34,3 +34,60 @@ func StatementAnnotations(s MicroflowStatement) *ActivityAnnotations { ann, _ := f.Interface().(*ActivityAnnotations) return ann } + +// StatementBodies returns every nested statement list a microflow statement +// contains — an IF's two branches, a CASE's arms, a loop body, an ON ERROR +// handler's body — so a check that has to span the whole flow can recurse +// without a type switch that goes stale. +// +// Reflective for the same reason as StatementAnnotations: a hand-written switch +// silently skips the statement type added after it was written, and the callers +// here are looking for something that would otherwise be missed entirely. +// TestStatementBodiesReachesEveryNestedBody pins the coverage. +func StatementBodies(s MicroflowStatement) [][]MicroflowStatement { + if s == nil { + return nil + } + v := reflect.ValueOf(s) + for v.Kind() == reflect.Ptr { + if v.IsNil() { + return nil + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return nil + } + + var out [][]MicroflowStatement + stmtSlice := reflect.TypeOf([]MicroflowStatement(nil)) + for i := 0; i < v.NumField(); i++ { + f := v.Field(i) + switch { + case f.Type() == stmtSlice: + if f.Len() > 0 { + out = append(out, f.Interface().([]MicroflowStatement)) + } + case f.Kind() == reflect.Ptr && f.Type() == reflect.TypeOf((*ErrorHandlingClause)(nil)): + if !f.IsNil() { + if body := f.Interface().(*ErrorHandlingClause).Body; len(body) > 0 { + out = append(out, body) + } + } + case f.Kind() == reflect.Slice: + // Case arms: []EnumSplitCase, []InheritanceSplitCase — each element + // is a struct with its own Body. + for j := 0; j < f.Len(); j++ { + el := f.Index(j) + if el.Kind() != reflect.Struct { + break + } + body := el.FieldByName("Body") + if body.IsValid() && body.Type() == stmtSlice && body.Len() > 0 { + out = append(out, body.Interface().([]MicroflowStatement)) + } + } + } + } + return out +} diff --git a/mdl/ast/annotations_coverage_test.go b/mdl/ast/annotations_coverage_test.go new file mode 100644 index 0000000000..98de9de4f5 --- /dev/null +++ b/mdl/ast/annotations_coverage_test.go @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +import ( + "go/ast" + "go/parser" + "go/token" + "reflect" + "testing" +) + +// StatementAnnotations and StatementBodies both read a statement by REFLECTION +// rather than through a type switch, on the argument that a switch silently +// skips the type added after it was written. That argument is only as good as +// the field-shape assumptions the reflection makes, and those are what these +// tests pin — by reading this package's own source, so a new statement type is +// covered the moment it is declared rather than when someone remembers to +// extend a list. + +// astStructFields parses the package source and yields every struct type with +// its fields. Source-driven because Go cannot enumerate the types implementing +// an interface at runtime, and a hand-maintained list here would have exactly +// the staleness problem the reflective readers exist to avoid. +func astStructFields(t *testing.T) map[string][]*ast.Field { + t.Helper() + fset := token.NewFileSet() + pkgs, err := parser.ParseDir(fset, ".", nil, 0) + if err != nil { + t.Fatalf("parsing the ast package source: %v", err) + } + out := map[string][]*ast.Field{} + for _, pkg := range pkgs { + for _, file := range pkg.Files { + ast.Inspect(file, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok { + return true + } + st, ok := ts.Type.(*ast.StructType) + if !ok || st.Fields == nil { + return true + } + out[ts.Name.Name] = st.Fields.List + return true + }) + } + } + if len(out) == 0 { + t.Fatal("no structs found — the source scan is not looking where it thinks it is") + } + return out +} + +func fieldTypeString(f *ast.Field) string { + switch tp := f.Type.(type) { + case *ast.Ident: + return tp.Name + case *ast.StarExpr: + if id, ok := tp.X.(*ast.Ident); ok { + return "*" + id.Name + } + case *ast.ArrayType: + if id, ok := tp.Elt.(*ast.Ident); ok { + return "[]" + id.Name + } + } + return "" +} + +// TestEveryAnnotatedStatementIsReachable pins the assumption StatementAnnotations +// rests on: the annotations of a statement are held in a field literally named +// `Annotations` of type *ActivityAnnotations. A statement type that spelled it +// any other way would have its annotations silently dropped — which is the very +// failure the reflective read was introduced to prevent (upstream #884). +func TestEveryAnnotatedStatementIsReachable(t *testing.T) { + for name, fields := range astStructFields(t) { + for _, f := range fields { + if fieldTypeString(f) != "*ActivityAnnotations" { + continue + } + if len(f.Names) != 1 || f.Names[0].Name != "Annotations" { + got := "embedded" + if len(f.Names) == 1 { + got = f.Names[0].Name + } + t.Errorf("%s holds its *ActivityAnnotations in a field named %q; "+ + "StatementAnnotations looks up \"Annotations\" by name, so this "+ + "statement's annotations are silently dropped", name, got) + } + } + } +} + +// TestStatementBodiesReachesEveryNestedBody pins the two assumptions +// StatementBodies rests on. The first — that a nested statement list is a field +// of type []MicroflowStatement — is checked structurally by the walker itself. +// The second is the fragile one: for a CASE ARM (a struct in a slice, like +// EnumSplitCase), the walker finds the arm's statements by looking up a field +// named `Body`. An arm that called it `Statements` would take its whole branch +// out of every whole-flow check with nothing to notice. +func TestStatementBodiesReachesEveryNestedBody(t *testing.T) { + structs := astStructFields(t) + for name, fields := range structs { + for _, f := range fields { + // A slice of some other struct declared in this package — the case-arm + // shape. If that struct holds statements, they must be in `Body`. + elem := fieldTypeString(f) + if len(elem) < 3 || elem[:2] != "[]" { + continue + } + armFields, ok := structs[elem[2:]] + if !ok { + continue + } + var stmtField string + for _, af := range armFields { + if fieldTypeString(af) == "[]MicroflowStatement" && len(af.Names) == 1 { + stmtField = af.Names[0].Name + } + } + if stmtField != "" && stmtField != "Body" { + t.Errorf("%s.%s is a slice of %s, whose statements live in %q; "+ + "StatementBodies looks up \"Body\", so that branch is invisible "+ + "to every check that walks the whole flow", + name, fieldName(f), elem[2:], stmtField) + } + } + } +} + +func fieldName(f *ast.Field) string { + if len(f.Names) == 1 { + return f.Names[0].Name + } + return "" +} + +// The behavioural half: a statement nesting every shape the walker handles must +// yield every one of its bodies. The control is the count — dropping any single +// arm of the switch in StatementBodies takes this below 5. +func TestStatementBodies_YieldsEveryShape(t *testing.T) { + mark := func(tag string) []MicroflowStatement { + return []MicroflowStatement{&LogStmt{Message: &LiteralExpr{Value: tag, Kind: LiteralString}}} + } + + ifStmt := &IfStmt{ThenBody: mark("then"), ElseBody: mark("else")} + if got := len(StatementBodies(ifStmt)); got != 2 { + t.Errorf("IfStmt: got %d bodies, want 2 (then, else)", got) + } + + enum := &EnumSplitStmt{ + Cases: []EnumSplitCase{{Value: "A", Body: mark("a")}, {Value: "B", Body: mark("b")}}, + ElseBody: mark("else"), + } + if got := len(StatementBodies(enum)); got != 3 { + t.Errorf("EnumSplitStmt: got %d bodies, want 3 (two arms + else)", got) + } + + declare := &DeclareStmt{ + Variable: "X", + ErrorHandling: &ErrorHandlingClause{Type: ErrorHandlingCustom, Body: mark("handler")}, + } + if got := len(StatementBodies(declare)); got != 1 { + t.Errorf("DeclareStmt with ON ERROR: got %d bodies, want 1 (the handler)", got) + } + + loop := &LoopStmt{LoopVariable: "Item", ListVariable: "Items", Body: mark("loop")} + if got := len(StatementBodies(loop)); got != 1 { + t.Errorf("LoopStmt: got %d bodies, want 1", got) + } + + // A statement with nothing nested must yield nothing, or a caller recursing + // on the result would never terminate. + if got := StatementBodies(&ReturnStmt{}); got != nil { + t.Errorf("ReturnStmt: got %v, want no bodies", got) + } + if got := StatementBodies(nil); got != nil { + t.Errorf("nil: got %v, want no bodies", got) + } + var typed *LogStmt + if got := StatementBodies(typed); got != nil { + t.Errorf("typed nil: got %v, want no bodies", got) + } +} + +// The reflective readers must not be fooled by a non-struct or a nil interior. +func TestStatementAnnotations_NilSafety(t *testing.T) { + if got := StatementAnnotations(nil); got != nil { + t.Errorf("nil: got %v", got) + } + var typed *LogStmt + if got := StatementAnnotations(typed); got != nil { + t.Errorf("typed nil: got %v", got) + } + stmt := &LogStmt{Annotations: &ActivityAnnotations{Caption: "c"}} + if got := StatementAnnotations(stmt); got == nil || got.Caption != "c" { + t.Errorf("got %v, want the annotations", got) + } + // Guards the field-name lookup against a same-named field of another type. + if reflect.TypeOf(ActivityAnnotations{}).Kind() != reflect.Struct { + t.Fatal("ActivityAnnotations is no longer a struct") + } +} diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index 150e1cb983..ed073e20ca 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -280,13 +280,18 @@ type FlowAnchors struct { // ActivityAnnotations holds metadata annotations for microflow activities. // These are emitted as @position, @caption, @color, @annotation, @excluded, @anchor lines in MDL. type ActivityAnnotations struct { - Position *Position // @position(x, y) - Caption string // @caption 'text' - Color string // @color Green - AnnotationText string // @annotation 'text' - FreeAnnotations []string // Multiple free-floating @annotation lines in source order - Excluded bool // @excluded - Anchor *FlowAnchors // @anchor(from: X, to: Y) — anchors of the flow leaving this statement + Position *Position // @position(x, y) + Caption string // @caption 'text' + Color string // @color Green + // Notes are the @annotation lines attached to this statement, in source + // order. A SLICE, not one string: see MicroflowAnnotation. + Notes []MicroflowAnnotation + + // FreeNotes are @annotation lines that stand on their own — a note on the + // canvas wired to nothing. + FreeNotes []MicroflowAnnotation + Excluded bool // @excluded + Anchor *FlowAnchors // @anchor(from: X, to: Y) — anchors of the flow leaving this statement // Split-specific anchors for IF statements. When the statement is not an // IF these remain nil. The grammar accepts them on IfStmt only: @@ -340,6 +345,12 @@ type ActivityAnnotations struct { // than silently straightening the edge. InvalidCurves []string + // InvalidNotes holds the raw text of any `@annotation(...)` parameter + // the visitor could not use — an unknown key, or a malformed `position:`/`size:` + // pair — so validation can refuse it. Dropping it would lose the note + // itself, not just the parameter. + InvalidNotes []string + // UnknownNames holds annotation names the visitor did not recognise, in // source order, so validation can refuse them. // @@ -352,6 +363,44 @@ type ActivityAnnotations struct { UnknownNames []string } +// MicroflowAnnotation is one `@annotation` line — the yellow note Studio Pro +// draws beside an activity. +// +// In Mendix's model a note is a NODE with edges (`Microflows$Annotation` joined +// to activities by `Microflows$AnnotationFlow`), not a property of the activity +// it documents: one note can be wired to several activities, and several notes +// to one activity. MDL modelled it as a single string per activity, which lost +// both directions — a shared note came back copied once per target, and a +// second note on one activity overwrote the first, silently +// (mendixlabs/mxcli#1077). Hence a slice, and hence Label. +type MicroflowAnnotation struct { + // Label is the `id:` in `@annotation(id: n1, text: '…')`. It exists only so + // a later `@annotation(id: n1)` can attach the SAME note to another + // activity instead of creating a second one. It is scoped to the flow being + // authored and is NOT stored in the model — the describer re-derives labels + // from scratch, so they are stable across a round trip by construction + // rather than by being remembered. + Label string + + // Text is the note's caption. Empty on a pure reference + // (`@annotation(id: n1)`), which attaches a note already declared above. + Text string + + // Position and Size are the note's own canvas geometry, which Mendix stores per + // annotation and MDL had no way to spell. Nil means "let the writer place + // it" — see defaultAnnotationGeometry in mdl/executor, which the builder and + // the describer both consult so a round trip need not spell out a position + // that can be re-derived. + Position *Position + Size *BoxSize +} + +// BoxSize is a width/height pair in canvas pixels. +type BoxSize struct { + Width int + Height int +} + // FlowCurve is the pair of bezier control vectors on a sequence flow. Either end // may be nil, which leaves that end straight. type FlowCurve struct { diff --git a/mdl/executor/cmd_microflows_builder.go b/mdl/executor/cmd_microflows_builder.go index 68e25ec3ad..94c53323d3 100644 --- a/mdl/executor/cmd_microflows_builder.go +++ b/mdl/executor/cmd_microflows_builder.go @@ -19,15 +19,22 @@ type flowBuilder struct { objects []microflows.MicroflowObject flows []*microflows.SequenceFlow annotationFlows []*microflows.AnnotationFlow - posX int - posY int - baseY int // Base Y position (for returning after ELSE branches) - spacing int - returnValue string // Return value expression for RETURN statement (used by buildFlowGraph final EndEvent) - returnType *ast.MicroflowReturnType - endsWithReturn bool // True if the flow already ends with EndEvent(s) from RETURN statements - lastReturnEndID model.ID // Last explicit RETURN EndEvent, used as a fallback error-handler target - varTypes map[string]string // Variable name -> entity qualified name (for CHANGE statements) + + // annotationsByLabel resolves `@annotation(id: n1)` back to the Annotation + // its first mention created, so a note wired to several activities is ONE + // object with several flows — the shape Mendix stores and the shape MDL + // could not previously express (#1077). Scoped to this flow build; labels + // are not stored in the model. + annotationsByLabel map[string]*microflows.Annotation + posX int + posY int + baseY int // Base Y position (for returning after ELSE branches) + spacing int + returnValue string // Return value expression for RETURN statement (used by buildFlowGraph final EndEvent) + returnType *ast.MicroflowReturnType + endsWithReturn bool // True if the flow already ends with EndEvent(s) from RETURN statements + lastReturnEndID model.ID // Last explicit RETURN EndEvent, used as a fallback error-handler target + varTypes map[string]string // Variable name -> entity qualified name (for CHANGE statements) // generatedVars holds the output-variable names minted for an unassigned // CREATE. Kept apart from varTypes so a generated name is never referenceable // from the script, while still reserving the name against a second create of diff --git a/mdl/executor/cmd_microflows_builder_annotations.go b/mdl/executor/cmd_microflows_builder_annotations.go index cac3016f3e..c2aff73b2a 100644 --- a/mdl/executor/cmd_microflows_builder_annotations.go +++ b/mdl/executor/cmd_microflows_builder_annotations.go @@ -128,11 +128,11 @@ func (fb *flowBuilder) mergeStatementAnnotations(stmt ast.MicroflowStatement) { if ann.Color != "" { fb.pendingAnnotations.Color = ann.Color } - if ann.AnnotationText != "" { - fb.pendingAnnotations.AnnotationText = ann.AnnotationText + if len(ann.Notes) > 0 { + fb.pendingAnnotations.Notes = append(fb.pendingAnnotations.Notes, ann.Notes...) } - if len(ann.FreeAnnotations) > 0 { - fb.pendingAnnotations.FreeAnnotations = append(fb.pendingAnnotations.FreeAnnotations, ann.FreeAnnotations...) + if len(ann.FreeNotes) > 0 { + fb.pendingAnnotations.FreeNotes = append(fb.pendingAnnotations.FreeNotes, ann.FreeNotes...) } if ann.Anchor != nil { fb.pendingAnnotations.Anchor = ann.Anchor @@ -206,9 +206,10 @@ func (fb *flowBuilder) applyAnnotations(activityID model.ID, ann *ast.ActivityAn } } - // @annotation — attach an annotation object - if ann.AnnotationText != "" { - fb.attachAnnotation(ann.AnnotationText, activityID) + // @annotation — attach the notes. All of them: an activity can carry more + // than one, and keeping only the last silently deleted the others (#1077). + for i, note := range ann.Notes { + fb.attachAnnotation(note, activityID, i) } } @@ -314,46 +315,124 @@ func (fb *flowBuilder) addErrorEvent() model.ID { return errorEvent.ID } -// attachAnnotation creates an Annotation object positioned above the given activity -// and connects them with an AnnotationFlow. -func (fb *flowBuilder) attachAnnotation(text string, activityID model.ID) { - // Find the activity's position to place annotation above it - var actX, actY int +// DefaultAnnotationSize is the note box Studio Pro creates, and what the writer +// uses when the script does not say (`@annotation 'text'` with no `size:`). +var DefaultAnnotationSize = model.Size{Width: 200, Height: 50} + +// defaultAnnotationGeometry is where an unplaced note goes: above the activity it +// documents, stacked upwards when several share one activity so the boxes do not +// land on top of each other. +// +// The DESCRIBER calls this too, to decide whether to emit `position:`/`size:` at +// all — +// a value the writer re-derives is omitted, which is what keeps a round-tripped +// note on the short `@annotation 'text'` form and makes only a hand-placed note +// pay for its geometry. +// +// Both sides MUST go through here. Two copies of the formula drift, and the +// failure is silent in the worst way: the describer omits a position the builder +// then re-derives differently, so the note creeps further on every round trip. +// TestAnnotationGeometryDefaultIsSharedByBothSides pins that. +func defaultAnnotationGeometry(activityPos model.Point, index int) (model.Point, model.Size) { + return model.Point{X: activityPos.X, Y: activityPos.Y - 100 - index*(DefaultAnnotationSize.Height+10)}, DefaultAnnotationSize +} + +// attachAnnotation attaches one note to an activity. +// +// A LABELLED note is created once and reused: `@annotation(id: n1, text: '…')` +// followed by `@annotation(id: n1)` on another activity yields ONE Annotation +// with two AnnotationFlows, which is how Mendix stores a note wired to several +// activities. Minting a fresh Annotation per mention is what duplicated the +// reporter's note on every round trip (#1077). +// +// index is the note's ordinal among those attached to this activity, used only +// to place unpositioned notes so they stack instead of overlapping. +func (fb *flowBuilder) attachAnnotation(note ast.MicroflowAnnotation, activityID model.ID, index int) { + if note.Label != "" { + if existing, ok := fb.annotationsByLabel[note.Label]; ok { + if note.Text != "" && note.Text != existing.Caption { + fb.addError("annotation id '%s' is declared twice with different text (%q, then %q) — "+ + "an id names ONE note; drop the id from the second one to make it a separate note", + note.Label, existing.Caption, note.Text) + return + } + fb.linkAnnotation(existing.ID, activityID) + return + } + if note.Text == "" { + fb.addError("@annotation(id: %s) refers to an annotation that has not been declared — "+ + "the first mention must carry the text, as @annotation(id: %s, text: '…')", + note.Label, note.Label) + return + } + } + + var activityPos model.Point for _, obj := range fb.objects { if obj.GetID() == activityID { - pos := obj.GetPosition() - actX = pos.X - actY = pos.Y + activityPos = obj.GetPosition() break } } + pos, size := defaultAnnotationGeometry(activityPos, index) + if note.Position != nil { + pos = model.Point{X: note.Position.X, Y: note.Position.Y} + } + if note.Size != nil { + size = model.Size{Width: note.Size.Width, Height: note.Size.Height} + } + + fb.linkAnnotation(fb.newAnnotation(note, pos, size).ID, activityID) +} +// attachFreeAnnotation creates a free-floating Annotation not connected to any +// activity. A label on one is accepted and reused, so a note can be shared +// between the canvas and an activity. +func (fb *flowBuilder) attachFreeAnnotation(note ast.MicroflowAnnotation) { + if note.Label != "" { + if _, ok := fb.annotationsByLabel[note.Label]; ok { + // Already created; a free mention adds no flow, so there is + // nothing left to do. + return + } + } + pos := model.Point{X: fb.posX, Y: fb.posY - 100} + if note.Position != nil { + pos = model.Point{X: note.Position.X, Y: note.Position.Y} + } + size := DefaultAnnotationSize + if note.Size != nil { + size = model.Size{Width: note.Size.Width, Height: note.Size.Height} + } + fb.newAnnotation(note, pos, size) +} + +// newAnnotation creates the Annotation object and, when the note is labelled, +// records it so a later mention of the same id attaches to THIS one instead of +// creating another. +func (fb *flowBuilder) newAnnotation(note ast.MicroflowAnnotation, pos model.Point, size model.Size) *microflows.Annotation { annotation := µflows.Annotation{ BaseMicroflowObject: microflows.BaseMicroflowObject{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - Position: model.Point{X: actX, Y: actY - 100}, - Size: model.Size{Width: 200, Height: 50}, + Position: pos, + Size: size, }, - Caption: text, + Caption: note.Text, } fb.objects = append(fb.objects, annotation) + if note.Label != "" { + if fb.annotationsByLabel == nil { + fb.annotationsByLabel = map[string]*microflows.Annotation{} + } + fb.annotationsByLabel[note.Label] = annotation + } + return annotation +} +func (fb *flowBuilder) linkAnnotation(annotationID, activityID model.ID) { fb.annotationFlows = append(fb.annotationFlows, µflows.AnnotationFlow{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - OriginID: annotation.ID, + OriginID: annotationID, DestinationID: activityID, }) } - -// attachFreeAnnotation creates a free-floating Annotation not connected to any activity. -func (fb *flowBuilder) attachFreeAnnotation(text string) { - annotation := µflows.Annotation{ - BaseMicroflowObject: microflows.BaseMicroflowObject{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - Position: model.Point{X: fb.posX, Y: fb.posY - 100}, - Size: model.Size{Width: 200, Height: 50}, - }, - Caption: text, - } - fb.objects = append(fb.objects, annotation) -} diff --git a/mdl/executor/cmd_microflows_builder_annotations_test.go b/mdl/executor/cmd_microflows_builder_annotations_test.go index 4bf2a8044d..c8adc52599 100644 --- a/mdl/executor/cmd_microflows_builder_annotations_test.go +++ b/mdl/executor/cmd_microflows_builder_annotations_test.go @@ -146,8 +146,8 @@ func TestIfAnnotationStaysWithCorrectSplit(t *testing.T) { &ast.ReturnStmt{Value: &ast.LiteralExpr{Value: false, Kind: ast.LiteralBoolean}}, }, Annotations: &ast.ActivityAnnotations{ - Caption: "Right format?", - AnnotationText: "Inner IF note", + Caption: "Right format?", + Notes: []ast.MicroflowAnnotation{{Text: "Inner IF note"}}, }, } outerIf := &ast.IfStmt{ @@ -161,8 +161,8 @@ func TestIfAnnotationStaysWithCorrectSplit(t *testing.T) { &ast.ReturnStmt{Value: &ast.LiteralExpr{Value: false, Kind: ast.LiteralBoolean}}, }, Annotations: &ast.ActivityAnnotations{ - Caption: "String not empty?", - AnnotationText: "Outer IF note", + Caption: "String not empty?", + Notes: []ast.MicroflowAnnotation{{Text: "Outer IF note"}}, }, } @@ -228,7 +228,7 @@ func TestLoopBodyIfAnnotationPromotedToParentFlows(t *testing.T) { &ast.LogStmt{Level: ast.LogInfo, Message: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "active"}}, }, Annotations: &ast.ActivityAnnotations{ - AnnotationText: "Nested decision note", + Notes: []ast.MicroflowAnnotation{{Text: "Nested decision note"}}, }, } loop := &ast.LoopStmt{ @@ -263,7 +263,7 @@ func TestLoopBodyIfAnnotationPromotedToParentFlows(t *testing.T) { } annotations := buildAnnotationsByTarget(oc) - if got := annotations[splitID]; len(got) != 1 || got[0] != "Nested decision note" { + if got := annotations.byTarget[splitID]; len(got) != 1 || got[0].Caption != "Nested decision note" { t.Fatalf("annotations for nested split = %#v, want Nested decision note", got) } } @@ -365,8 +365,8 @@ func TestFreeAnnotationBeforePositionStaysUnattached(t *testing.T) { Level: ast.LogInfo, Message: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "message"}, Annotations: &ast.ActivityAnnotations{ - FreeAnnotations: []string{"free synthetic note"}, - Position: &ast.Position{X: 120, Y: 240}, + FreeNotes: []ast.MicroflowAnnotation{{Text: "free synthetic note"}}, + Position: &ast.Position{X: 120, Y: 240}, }, }, } @@ -375,14 +375,14 @@ func TestFreeAnnotationBeforePositionStaysUnattached(t *testing.T) { oc := fb.buildFlowGraph(body, nil) freeAnnotations := collectFreeAnnotations(oc) - if len(freeAnnotations) != 1 || freeAnnotations[0] != "free synthetic note" { + if len(freeAnnotations) != 1 || freeAnnotations[0].Caption != "free synthetic note" { t.Fatalf("free annotations = %#v, want one free note", freeAnnotations) } attached := buildAnnotationsByTarget(oc) - for activityID, captions := range attached { - for _, caption := range captions { - if caption == "free synthetic note" { + for activityID, notes := range attached.byTarget { + for _, n := range notes { + if n.Caption == "free synthetic note" { t.Fatalf("free note was attached to activity %s", activityID) } } @@ -395,8 +395,8 @@ func TestMultipleFreeAnnotationsBeforePositionStayUnattached(t *testing.T) { Level: ast.LogInfo, Message: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "message"}, Annotations: &ast.ActivityAnnotations{ - FreeAnnotations: []string{"first free note", "second free note"}, - Position: &ast.Position{X: 120, Y: 240}, + FreeNotes: []ast.MicroflowAnnotation{{Text: "first free note"}, {Text: "second free note"}}, + Position: &ast.Position{X: 120, Y: 240}, }, }, } @@ -410,8 +410,8 @@ func TestMultipleFreeAnnotationsBeforePositionStayUnattached(t *testing.T) { t.Fatalf("free annotations = %#v, want %#v", freeAnnotations, want) } for i, wantText := range want { - if freeAnnotations[i] != wantText { - t.Fatalf("free annotation %d = %q, want %q", i, freeAnnotations[i], wantText) + if freeAnnotations[i].Caption != wantText { + t.Fatalf("free annotation %d = %q, want %q", i, freeAnnotations[i].Caption, wantText) } } } @@ -457,7 +457,7 @@ func TestIfBranchActionAnnotationStaysWithAction(t *testing.T) { Level: ast.LogInfo, Message: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "branch"}, Annotations: &ast.ActivityAnnotations{ - AnnotationText: "Branch note", + Notes: []ast.MicroflowAnnotation{{Text: "Branch note"}}, }, }, }, @@ -482,7 +482,7 @@ func TestIfBranchActionAnnotationStaysWithAction(t *testing.T) { } attached := buildAnnotationsByTarget(oc) - if got := attached[logID]; len(got) != 1 || got[0] != "Branch note" { + if got := attached.byTarget[logID]; len(got) != 1 || got[0].Caption != "Branch note" { t.Fatalf("branch log annotations = %#v, want [Branch note]", got) } } diff --git a/mdl/executor/cmd_microflows_builder_control.go b/mdl/executor/cmd_microflows_builder_control.go index 5ad2e673a0..7fa377c0be 100644 --- a/mdl/executor/cmd_microflows_builder_control.go +++ b/mdl/executor/cmd_microflows_builder_control.go @@ -614,6 +614,19 @@ func (fb *flowBuilder) addLoopStatement(s *ast.LoopStmt) model.ID { hierarchy: fb.hierarchy, // Share hierarchy restServices: fb.restServices, // Share REST services for parameter classification isNanoflow: fb.isNanoflow, + // Share the note registry, so a note declared outside the loop and + // referenced on a body activity attaches to the SAME Annotation rather + // than being refused. The describer emits exactly that (its label state + // is shared across the loop overlay), so a builder that refused it would + // reject its own DESCRIBE output. + // + // The resulting shape — Annotation in the parent collection, both + // AnnotationFlows in the parent collection, one destination inside the + // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only + // oracle available here; Studio Pro is stricter in general, though this + // is the same split mxcli already ships for an ordinary loop-body note, + // whose flow is hoisted to the parent a few lines below. (#1077) + annotationsByLabel: fb.annotationsByLabel, } // Process loop body statements and connect them with flows. @@ -701,6 +714,13 @@ func (fb *flowBuilder) addLoopStatement(s *ast.LoopStmt) model.ID { // This is how Mendix stores them - all flows at the microflow level fb.flows = append(fb.flows, loopBuilder.flows...) fb.annotationFlows = append(fb.annotationFlows, loopBuilder.annotationFlows...) + // Refusals raised in the body were collected into the sub-builder and + // dropped on the floor, so exec reported success on a flow it had silently + // written wrong (#1077). + fb.errors = append(fb.errors, loopBuilder.errors...) + if fb.annotationsByLabel == nil { + fb.annotationsByLabel = loopBuilder.annotationsByLabel + } // Re-apply this loop's own annotations now that its activity exists. if savedLoopAnnotations != nil { @@ -932,6 +952,19 @@ func (fb *flowBuilder) addWhileStatement(s *ast.WhileStmt) model.ID { hierarchy: fb.hierarchy, restServices: fb.restServices, isNanoflow: fb.isNanoflow, + // Share the note registry, so a note declared outside the loop and + // referenced on a body activity attaches to the SAME Annotation rather + // than being refused. The describer emits exactly that (its label state + // is shared across the loop overlay), so a builder that refused it would + // reject its own DESCRIBE output. + // + // The resulting shape — Annotation in the parent collection, both + // AnnotationFlows in the parent collection, one destination inside the + // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only + // oracle available here; Studio Pro is stricter in general, though this + // is the same split mxcli already ships for an ordinary loop-body note, + // whose flow is hoisted to the parent a few lines below. (#1077) + annotationsByLabel: fb.annotationsByLabel, } // Body bookkeeping is addLoopStatement's, verbatim: a WHILE body is a loop @@ -1009,6 +1042,13 @@ func (fb *flowBuilder) addWhileStatement(s *ast.WhileStmt) model.ID { fb.objects = append(fb.objects, loop) fb.flows = append(fb.flows, loopBuilder.flows...) fb.annotationFlows = append(fb.annotationFlows, loopBuilder.annotationFlows...) + // Refusals raised in the body were collected into the sub-builder and + // dropped on the floor, so exec reported success on a flow it had silently + // written wrong (#1077). + fb.errors = append(fb.errors, loopBuilder.errors...) + if fb.annotationsByLabel == nil { + fb.annotationsByLabel = loopBuilder.annotationsByLabel + } if savedWhileAnnotations != nil { fb.applyAnnotations(loop.ID, savedWhileAnnotations) diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 5079368667..330b288d48 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -721,6 +721,11 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in hierarchy: fb.hierarchy, restServices: fb.restServices, isNanoflow: fb.isNanoflow, + // A handler's activities are merged into the PARENT's object collection + // below, so a note declared outside the handler and referenced inside it + // (or the reverse) lands in one collection — sharing the registry is + // sound here in a way it is not across a loop boundary (#1077). + annotationsByLabel: fb.annotationsByLabel, } var lastErrID model.ID @@ -751,9 +756,19 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in } } - // Append error handler objects and flows to the main builder + // Append error handler objects and flows to the main builder. + // + // annotationFlows and errors are part of that: without them a note written + // inside `on error { … }` arrived as an Annotation with no edge — a + // free-floating sticky note instead of one attached to the activity — and a + // refusal raised in the handler body never reached the caller (#1077). fb.objects = append(fb.objects, errBuilder.objects...) fb.flows = append(fb.flows, errBuilder.flows...) + fb.annotationFlows = append(fb.annotationFlows, errBuilder.annotationFlows...) + fb.errors = append(fb.errors, errBuilder.errors...) + if fb.annotationsByLabel == nil { + fb.annotationsByLabel = errBuilder.annotationsByLabel + } // If the error handler ends with RAISE ERROR or RETURN, it terminates there. // Otherwise, return the last activity ID so caller can create a merge. diff --git a/mdl/executor/cmd_microflows_builder_graph.go b/mdl/executor/cmd_microflows_builder_graph.go index 86dab92172..cff0617ebe 100644 --- a/mdl/executor/cmd_microflows_builder_graph.go +++ b/mdl/executor/cmd_microflows_builder_graph.go @@ -148,11 +148,12 @@ func (fb *flowBuilder) buildFlowGraph(stmts []ast.MicroflowStatement, returns *a // Free annotations are standalone Annotation objects. Flush them before // creating the activity so they do not get attached to it; buildFlowGraph // has a final leftover flush for annotations with no following activity. - for _, text := range fb.pendingAnnotations.FreeAnnotations { - fb.attachFreeAnnotation(text) + for _, note := range fb.pendingAnnotations.FreeNotes { + fb.attachFreeAnnotation(note) } - if fb.pendingAnnotations.AnnotationText != "" { - fb.attachFreeAnnotation(fb.pendingAnnotations.AnnotationText) + // An attached note with no activity left to attach to is a free one. + for _, note := range fb.pendingAnnotations.Notes { + fb.attachFreeAnnotation(note) } fb.pendingAnnotations = nil } @@ -538,10 +539,10 @@ func (fb *flowBuilder) addStatement(stmt ast.MicroflowStatement) model.ID { fb.posY = fb.pendingAnnotations.Position.Y } if fb.pendingAnnotations != nil { - for _, text := range fb.pendingAnnotations.FreeAnnotations { - fb.attachFreeAnnotation(text) + for _, note := range fb.pendingAnnotations.FreeNotes { + fb.attachFreeAnnotation(note) } - fb.pendingAnnotations.FreeAnnotations = nil + fb.pendingAnnotations.FreeNotes = nil } switch s := stmt.(type) { diff --git a/mdl/executor/cmd_microflows_describe_loop_annotation_test.go b/mdl/executor/cmd_microflows_describe_loop_annotation_test.go index 8997da796f..11770cb0cc 100644 --- a/mdl/executor/cmd_microflows_describe_loop_annotation_test.go +++ b/mdl/executor/cmd_microflows_describe_loop_annotation_test.go @@ -51,7 +51,12 @@ func TestEmitLoopBody_AnnotationNeverBecomesTheBodyStart(t *testing.T) { if !strings.Contains(out, "in the loop") { t.Errorf("loop body was dropped — the activity is absent from describe output.\ngot:\n%s", out) } - if !strings.Contains(out, "@annotation 'explains the step'") { + // Matched on the text, not on a whole line: every case here places + // the note somewhere the writer would not have put it, so its + // position is spelled out and the long form is the correct emit + // (mendixlabs/mxcli#1077). What this test is about is that the note + // survives and does not become the body's first statement. + if !strings.Contains(out, "'explains the step'") { t.Errorf("the annotation itself was dropped.\ngot:\n%s", out) } }) @@ -139,7 +144,7 @@ func TestDescribeLoopBody_SurvivesAnnotationAtBuilderPositions(t *testing.T) { &ast.LogStmt{ Level: ast.LogInfo, Message: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "has name"}, - Annotations: &ast.ActivityAnnotations{AnnotationText: "note on statement"}, + Annotations: &ast.ActivityAnnotations{Notes: []ast.MicroflowAnnotation{{Text: "note on statement"}}}, }, }, }, diff --git a/mdl/executor/cmd_microflows_show_helpers.go b/mdl/executor/cmd_microflows_show_helpers.go index 5995e27b62..b4d98836ac 100644 --- a/mdl/executor/cmd_microflows_show_helpers.go +++ b/mdl/executor/cmd_microflows_show_helpers.go @@ -15,84 +15,207 @@ import ( "github.com/mendixlabs/mxcli/sdk/microflows" ) -// buildAnnotationsByTarget builds a map from activity ID to annotation captions. -// It joins AnnotationFlows (destination → activity) with Annotation objects (caption). -func buildAnnotationsByTarget(oc *microflows.MicroflowObjectCollection) map[model.ID][]string { - result := make(map[model.ID][]string) +// describedAnnotation is one Annotation object as the describer sees it: its +// text, its canvas geometry — and its IDENTITY, which is the part that used to +// be thrown away here. A note wired to N activities was filed under each target +// as a bare caption, so DESCRIBE emitted N copies and re-executing the output +// created N notes (mendixlabs/mxcli#1077). +type describedAnnotation struct { + ID model.ID + Caption string + Position model.Point + Size model.Size + + // Shared is true when more than one AnnotationFlow points at this note. + // Only a shared note needs a label, so the everyday `@annotation 'text'` + // form is left exactly as it was. + Shared bool +} + +// annotationEmitter renders the @annotation lines for a describe traversal. +// +// It is stateful on purpose: WHICH mention of a shared note carries the +// declaration and which are references depends on traversal order, and that is +// only known while walking. One emitter per describe call — never a package +// global, so concurrent describes (captureDescribeParallel) stay independent. +type annotationEmitter struct { + byTarget map[model.ID][]describedAnnotation + + // labels and nextLabel are SHARED by reference with any overlay emitter + // (see withOverlay), because a note may be declared in the parent + // collection and referenced from inside a loop body. + labels map[model.ID]string + nextLabel *int +} + +// buildAnnotationsByTarget joins AnnotationFlows (destination → activity) with +// the Annotation objects they originate from, keeping each note's identity so a +// shared one can be emitted once and referenced afterwards. +func buildAnnotationsByTarget(oc *microflows.MicroflowObjectCollection) *annotationEmitter { + zero := 0 + e := &annotationEmitter{ + byTarget: make(map[model.ID][]describedAnnotation), + labels: make(map[model.ID]string), + nextLabel: &zero, + } if oc == nil { - return result + return e } - annotCaptions := make(map[model.ID]string) - collectAnnotationCaptions(oc, annotCaptions) + notes := make(map[model.ID]*microflows.Annotation) + collectAnnotationObjects(oc, notes) + + targetCount := make(map[model.ID]int) + for _, af := range oc.AnnotationFlows { + targetCount[af.OriginID]++ + } - // Map each annotation flow's destination (the activity) to the annotation's caption for _, af := range oc.AnnotationFlows { - if caption, ok := annotCaptions[af.OriginID]; ok && caption != "" { - result[af.DestinationID] = append(result[af.DestinationID], caption) + note, ok := notes[af.OriginID] + if !ok || note.Caption == "" { + continue } + e.byTarget[af.DestinationID] = append(e.byTarget[af.DestinationID], describedAnnotation{ + ID: note.ID, + Caption: note.Caption, + Position: note.Position, + Size: note.Size, + Shared: targetCount[af.OriginID] > 1, + }) } - return result + return e } -func collectAnnotationCaptions(oc *microflows.MicroflowObjectCollection, captions map[model.ID]string) { +func collectAnnotationObjects(oc *microflows.MicroflowObjectCollection, out map[model.ID]*microflows.Annotation) { if oc == nil { return } for _, obj := range oc.Objects { if annot, ok := obj.(*microflows.Annotation); ok { - captions[annot.ID] = annot.Caption + out[annot.ID] = annot continue } if loop, ok := obj.(*microflows.LoopedActivity); ok { - collectAnnotationCaptions(loop.ObjectCollection, captions) + collectAnnotationObjects(loop.ObjectCollection, out) } } } -// mergeAnnotationsByTarget combines parent-level annotations with the -// loop-local overlay so each activity gets every caption that points at it, -// regardless of which collection the annotation flow lives in. +// withOverlay returns an emitter covering both this emitter's targets and the +// overlay's, sharing the label state by reference so a note declared in the +// parent collection and referenced inside a loop body keeps one label. // -// When one side is empty the function returns the other map by reference (no -// copy). The current callers — emitLoopBody passing a freshly built overlay, -// or a freshly inherited parent map — never mutate the result, so aliasing is -// safe. New callers that intend to mutate the result must copy first. -func mergeAnnotationsByTarget(base, overlay map[model.ID][]string) map[model.ID][]string { - if len(base) == 0 { +// Neither input map is mutated. When one side is empty the other's map is +// returned by reference (no copy), which the current callers — emitLoopBody +// with a freshly built overlay — never write to. +func (e *annotationEmitter) withOverlay(overlay *annotationEmitter) *annotationEmitter { + if e == nil { return overlay } - if len(overlay) == 0 { - return base + if overlay == nil || len(overlay.byTarget) == 0 { + return e } - merged := make(map[model.ID][]string, len(base)+len(overlay)) - for id, captions := range base { - merged[id] = captions + out := &annotationEmitter{labels: e.labels, nextLabel: e.nextLabel} + if len(e.byTarget) == 0 { + out.byTarget = overlay.byTarget + return out } - for id, captions := range overlay { - merged[id] = append(merged[id], captions...) + out.byTarget = make(map[model.ID][]describedAnnotation, len(e.byTarget)+len(overlay.byTarget)) + for id, notes := range e.byTarget { + out.byTarget[id] = notes } - return merged + for id, notes := range overlay.byTarget { + out.byTarget[id] = append(out.byTarget[id], notes...) + } + return out } -// collectFreeAnnotations returns captions for annotations not referenced by any AnnotationFlow. -func collectFreeAnnotations(oc *microflows.MicroflowObjectCollection) []string { +// labelFor returns the emitted label for a note and whether this is its first +// mention (the one that must carry the text). +func (e *annotationEmitter) labelFor(id model.ID) (label string, first bool) { + if existing, ok := e.labels[id]; ok { + return existing, false + } + // Lazily initialised so a hand-built emitter (tests construct one with just + // byTarget) cannot panic on a shared note. + if e.labels == nil { + e.labels = map[model.ID]string{} + } + if e.nextLabel == nil { + e.nextLabel = new(int) + } + *e.nextLabel++ + label = fmt.Sprintf("n%d", *e.nextLabel) + e.labels[id] = label + return label, true +} + +// lines renders the @annotation lines for one activity. +// +// The short form is kept wherever it is lossless: an unshared note sitting at +// the position and size the writer would re-derive emits `@annotation 'text'`, +// exactly as before. Only a note that is shared, or that has been moved or +// resized on the canvas, pays for the longer form — so this fix does not churn +// the output of every microflow that has a note in it. +func (e *annotationEmitter) lines(target model.ID, activityPos model.Point, indentStr string) []string { + if e == nil { + return nil + } + var out []string + for i, note := range e.byTarget[target] { + if note.Shared { + label, first := e.labelFor(note.ID) + if !first { + out = append(out, indentStr+fmt.Sprintf("@annotation(id: %s)", label)) + continue + } + // A shared note's geometry is always spelled out. Its default + // depends on which mention created it, and pinning that down would + // couple the two sides far more tightly than it is worth. + out = append(out, indentStr+fmt.Sprintf("@annotation(id: %s, text: %s, position: (%d, %d), size: (%d, %d))", + label, mdlQuote(note.Caption), note.Position.X, note.Position.Y, note.Size.Width, note.Size.Height)) + continue + } + + defPos, defSize := defaultAnnotationGeometry(activityPos, i) + var params []string + if note.Position != defPos { + params = append(params, fmt.Sprintf("position: (%d, %d)", note.Position.X, note.Position.Y)) + } + if note.Size != defSize && note.Size != (model.Size{}) { + params = append(params, fmt.Sprintf("size: (%d, %d)", note.Size.Width, note.Size.Height)) + } + if len(params) == 0 { + out = append(out, indentStr+fmt.Sprintf("@annotation %s", mdlQuote(note.Caption))) + continue + } + out = append(out, indentStr+fmt.Sprintf("@annotation(text: %s, %s)", + mdlQuote(note.Caption), strings.Join(params, ", "))) + } + return out +} + +// collectFreeAnnotations returns the annotations no AnnotationFlow points at — +// notes that sit on the canvas documenting the flow as a whole. +func collectFreeAnnotations(oc *microflows.MicroflowObjectCollection) []describedAnnotation { if oc == nil { return nil } - // Collect annotation IDs that are referenced by flows referencedAnnotations := make(map[model.ID]bool) for _, af := range oc.AnnotationFlows { referencedAnnotations[af.OriginID] = true } - var result []string + var result []describedAnnotation for _, obj := range oc.Objects { if annot, ok := obj.(*microflows.Annotation); ok { if !referencedAnnotations[annot.ID] && annot.Caption != "" { - result = append(result, annot.Caption) + result = append(result, describedAnnotation{ + ID: annot.ID, Caption: annot.Caption, + Position: annot.Position, Size: annot.Size, + }) } } } @@ -106,8 +229,15 @@ func prependFreeAnnotationLines(oc *microflows.MicroflowObjectCollection, activi } prefix := make([]string, 0, len(freeAnnots)) - for _, text := range freeAnnots { - prefix = append(prefix, fmt.Sprintf("@annotation %s", mdlQuote(text))) + for _, note := range freeAnnots { + // A free note is attached to nothing, so there is no activity position + // to derive its default from — the writer places it from wherever the + // cursor happens to be. Its geometry is therefore always spelled out. + line := fmt.Sprintf("@annotation(text: %s, position: (%d, %d)", mdlQuote(note.Caption), note.Position.X, note.Position.Y) + if note.Size != (model.Size{}) { + line += fmt.Sprintf(", size: (%d, %d)", note.Size.Width, note.Size.Height) + } + prefix = append(prefix, line+")") } return append(prefix, activityLines...) } @@ -630,7 +760,7 @@ func emitObjectAnnotations( obj microflows.MicroflowObject, lines *[]string, indentStr string, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, flowsByOrigin map[model.ID][]*microflows.SequenceFlow, flowsByDest map[model.ID][]*microflows.SequenceFlow, activityMap map[model.ID]microflows.MicroflowObject, @@ -672,11 +802,7 @@ func emitObjectAnnotations( } // @annotation (attached Annotation objects) - if annotationsByTarget != nil { - for _, caption := range annotationsByTarget[currentID] { - *lines = append(*lines, indentStr+fmt.Sprintf("@annotation %s", mdlQuote(caption))) - } - } + *lines = append(*lines, annotationsByTarget.lines(currentID, pos, indentStr)...) } // emitActivityStatement appends the formatted activity statement (with error handling) @@ -693,7 +819,7 @@ func emitActivityStatement( microflowNames map[model.ID]string, lines *[]string, indentStr string, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if stmt == "" { return @@ -716,7 +842,7 @@ func emitActivityStatement( // render it commented-out, so the artifact still shows what the model // holds. Guard-don't-drop, in a path that cannot round-trip. emitCommentedErrorHandler( - ctx, obj, flowsByOrigin, activityMap, entityNames, microflowNames, lines, indentStr) + ctx, obj, flowsByOrigin, activityMap, entityNames, microflowNames, lines, indentStr, annotationsByTarget) return } @@ -740,7 +866,7 @@ func emitActivityStatement( errStmts := collectErrorHandlerStatements( ctx, errorHandlerFlow.DestinationID, - activityMap, flowsByOrigin, entityNames, microflowNames, + activityMap, flowsByOrigin, entityNames, microflowNames, annotationsByTarget, ) stmtWithoutSemi := strings.TrimSuffix(strings.TrimSpace(stmt), ";") @@ -785,6 +911,7 @@ func emitCommentedErrorHandler( microflowNames map[model.ID]string, lines *[]string, indentStr string, + annotationsByTarget *annotationEmitter, ) { errorHandlerFlow := findErrorHandlerFlow(flowsByOrigin[obj.GetID()]) if errorHandlerFlow == nil { @@ -801,7 +928,7 @@ func emitCommentedErrorHandler( } errStmts := collectErrorHandlerStatements( - ctx, errorHandlerFlow.DestinationID, activityMap, flowsByOrigin, entityNames, microflowNames) + ctx, errorHandlerFlow.DestinationID, activityMap, flowsByOrigin, entityNames, microflowNames, annotationsByTarget) if len(errStmts) == 0 { *lines = append(*lines, indentStr+"-- "+suffix+" { };") return @@ -842,7 +969,7 @@ func traverseFlow( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if currentID == "" || visited[currentID] { return @@ -1033,7 +1160,7 @@ func traverseFlowUntilMerge( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if currentID == "" || currentID == mergeID || visited[currentID] { return @@ -1202,7 +1329,7 @@ func continueAfterSplitJoin( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if joinID == "" { return @@ -1232,7 +1359,7 @@ func continueAfterNestedSplitJoin( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if joinID == "" || joinID == parentMergeID { return @@ -1320,7 +1447,7 @@ func traverseLoopBody( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { // Loop bodies can contain the same structured control flow as top-level // microflows. Reuse the main traversal with a loop-local split/merge map so @@ -1342,13 +1469,13 @@ func emitLoopBody( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { if loop.ObjectCollection == nil || len(loop.ObjectCollection.Objects) == 0 { return } - loopAnnotationsByTarget := mergeAnnotationsByTarget(annotationsByTarget, buildAnnotationsByTarget(loop.ObjectCollection)) + loopAnnotationsByTarget := annotationsByTarget.withOverlay(buildAnnotationsByTarget(loop.ObjectCollection)) // Build a map of objects in the loop body loopActivityMap := make(map[model.ID]microflows.MicroflowObject) @@ -1564,7 +1691,7 @@ func emitEnumSplitStatement( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { indentStr := strings.Repeat(" ", indent) *lines = append(*lines, indentStr+"case $"+variable) @@ -1621,7 +1748,7 @@ func emitInheritanceSplitStatement( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { split, _ := activityMap[currentID].(*microflows.InheritanceSplit) if split == nil { @@ -2105,10 +2232,20 @@ func collectErrorHandlerStatements( flowsByOrigin map[model.ID][]*microflows.SequenceFlow, entityNames map[model.ID]string, microflowNames map[model.ID]string, + annotationsByTarget *annotationEmitter, ) []string { var statements []string visited := make(map[model.ID]bool) stopID := firstReachableErrorHandlerMerge(startID, activityMap, flowsByOrigin) + + // A note on a handler-body activity is emitted here or nowhere: this + // traversal is a second, smaller describer and the main one never reaches + // inside an `on error { … }` block. Without it the write path attaches the + // note and the read path drops it, which is the same round-trip loss #1077 + // is about, one nesting level down. + notes := func(obj microflows.MicroflowObject, indentStr string) { + statements = append(statements, annotationsByTarget.lines(obj.GetID(), obj.GetPosition(), indentStr)...) + } splitMergeMap := findErrorHandlerSplitMergePoints(ctx, activityMap, flowsByOrigin) var traverse func(id model.ID, boundary model.ID, indent int) @@ -2129,6 +2266,7 @@ func collectErrorHandlerStatements( if _, isSplit := obj.(*microflows.ExclusiveSplit); isSplit { stmt := formatActivity(ctx, obj, entityNames, microflowNames) if stmt != "" { + notes(obj, indentStr) statements = append(statements, indentStr+stmt) } nestedMergeID := splitMergeMap[id] @@ -2155,6 +2293,7 @@ func collectErrorHandlerStatements( } if stmt := formatActivity(ctx, obj, entityNames, microflowNames); stmt != "" { + notes(obj, indentStr) statements = append(statements, indentStr+stmt) } for _, flow := range findNormalFlows(flowsByOrigin[id]) { @@ -2229,7 +2368,7 @@ func (e *Executor) traverseFlow( indent int, sourceMap map[string]elkSourceRange, headerLineCount int, - annotationsByTarget map[model.ID][]string, + annotationsByTarget *annotationEmitter, ) { // Legacy wrapper — preserved for tests and unmigrated callers that don't // supply flowsByDest. Passing nil suppresses @anchor emission, matching @@ -2278,6 +2417,7 @@ func (e *Executor) collectErrorHandlerStatements( flowsByOrigin map[model.ID][]*microflows.SequenceFlow, entityNames map[model.ID]string, microflowNames map[model.ID]string, + annotationsByTarget *annotationEmitter, ) []string { - return collectErrorHandlerStatements(e.newExecContext(context.Background()), startID, activityMap, flowsByOrigin, entityNames, microflowNames) + return collectErrorHandlerStatements(e.newExecContext(context.Background()), startID, activityMap, flowsByOrigin, entityNames, microflowNames, annotationsByTarget) } diff --git a/mdl/executor/cmd_microflows_show_helpers_test.go b/mdl/executor/cmd_microflows_show_helpers_test.go index c2bb1ac506..750c7fe30d 100644 --- a/mdl/executor/cmd_microflows_show_helpers_test.go +++ b/mdl/executor/cmd_microflows_show_helpers_test.go @@ -143,8 +143,10 @@ func TestEmitObjectAnnotations_EscapesMultilineText(t *testing.T) { }, } - annotationsByTarget := map[model.ID][]string{ - mkID("act"): {"Note\nLine\tTabbed"}, + annotationsByTarget := &annotationEmitter{ + byTarget: map[model.ID][]describedAnnotation{ + mkID("act"): {{Caption: "Note\nLine\tTabbed", Position: mustDefaultAnnotationPos(model.Point{X: 100, Y: 200}, 0)}}, + }, } var lines []string @@ -210,7 +212,11 @@ func TestPrependFreeAnnotationLines_ModelAnnotationsStayFree(t *testing.T) { got := strings.Join(gotLines, "\n") want := strings.Join([]string{ - "@annotation 'free synthetic note'", + // A free note is wired to no activity, so there is no activity position + // for the writer to derive its place from — its own position is always + // emitted. This fixture sets none, hence (0, 0); a real canvas note + // carries the coordinates Studio Pro gave it (#1077). + "@annotation(text: 'free synthetic note', position: (0, 0))", "@position(100, 200)", "@annotation 'attached synthetic note'", "log info 'Synthetic' 'message';", @@ -542,3 +548,11 @@ func TestFormatErrorHandlingSuffix_RollbackIsNotEmitted(t *testing.T) { } } } + +// mustDefaultAnnotationPos is the position the writer would give an unplaced +// note, so this test exercises the SHORT emit form rather than accidentally +// asserting escaping on the parameterised one. +func mustDefaultAnnotationPos(activity model.Point, index int) model.Point { + pos, _ := defaultAnnotationGeometry(activity, index) + return pos +} diff --git a/mdl/executor/cmd_microflows_traverse_test.go b/mdl/executor/cmd_microflows_traverse_test.go index c7413c40d5..ec85ee049c 100644 --- a/mdl/executor/cmd_microflows_traverse_test.go +++ b/mdl/executor/cmd_microflows_traverse_test.go @@ -1003,10 +1003,8 @@ func TestTraverseFlow_LoopBodyUsesNestedAnnotationFlows(t *testing.T) { }, }, } - annotationsByTarget := mergeAnnotationsByTarget( - buildAnnotationsByTarget(µflows.MicroflowObjectCollection{}), - buildAnnotationsByTarget(loopObjects), - ) + annotationsByTarget := buildAnnotationsByTarget(µflows.MicroflowObjectCollection{}). + withOverlay(buildAnnotationsByTarget(loopObjects)) var lines []string e.traverseFlow( @@ -1025,7 +1023,10 @@ func TestTraverseFlow_LoopBodyUsesNestedAnnotationFlows(t *testing.T) { ) out := strings.Join(lines, "\n") - if !strings.Contains(out, "@annotation 'nested split note'") { + // The long form because this note sits at (1000, 100) while the split it + // documents is at (100, 100) — nowhere near where the writer would place an + // unpositioned note, so its position is spelled out rather than lost (#1077). + if !strings.Contains(out, "@annotation(text: 'nested split note', position: (1000, 100))") { t.Fatalf("expected nested loop annotation in output:\n%s", out) } } @@ -1055,7 +1056,7 @@ func TestCollectErrorHandlerStatements_Simple(t *testing.T) { mkID("err_log"): {mkFlow("err_log", "err_end")}, } - stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil) + stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil) if len(stmts) != 2 { t.Fatalf("expected 2 statements, got %d: %v", len(stmts), stmts) } @@ -1084,7 +1085,7 @@ func TestCollectErrorHandlerStatements_StopsAtMerge(t *testing.T) { mkID("merge"): {mkFlow("merge", "after")}, } - stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil) + stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil) // Should stop at merge, not include "after" if len(stmts) != 1 { t.Fatalf("expected 1 statement (stop at merge), got %d: %v", len(stmts), stmts) @@ -1117,7 +1118,7 @@ func TestCollectErrorHandlerStatements_StructuredIfEmitsEndIf(t *testing.T) { mkID("merge"): {mkFlow("merge", "after")}, } - stmts := e.collectErrorHandlerStatements(mkID("split"), activityMap, flowsByOrigin, nil, nil) + stmts := e.collectErrorHandlerStatements(mkID("split"), activityMap, flowsByOrigin, nil, nil, nil) got := strings.Join(stmts, "\n") assertContains(t, got, "if $latestHttpResponse != empty then") @@ -1131,7 +1132,7 @@ func TestCollectErrorHandlerStatements_StructuredIfEmitsEndIf(t *testing.T) { func TestCollectErrorHandlerStatements_EmptyID(t *testing.T) { e := newTestExecutor() - stmts := e.collectErrorHandlerStatements("", nil, nil, nil, nil) + stmts := e.collectErrorHandlerStatements("", nil, nil, nil, nil, nil) if len(stmts) != 0 { t.Errorf("expected 0 statements for empty ID, got %d", len(stmts)) } diff --git a/mdl/executor/microflow_annotation_sharing_test.go b/mdl/executor/microflow_annotation_sharing_test.go new file mode 100644 index 0000000000..d7e5bbe0e1 --- /dev/null +++ b/mdl/executor/microflow_annotation_sharing_test.go @@ -0,0 +1,542 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1077 — "roundtrip for annotation with multiple connections +// results in duplicate annotations". +// +// In Mendix a note is a NODE with edges: one Microflows$Annotation joined to any +// number of activities by Microflows$AnnotationFlow. MDL modelled it as one +// string per activity, and that lost BOTH directions of the relation: +// +// A. one note → N activities came back as N notes (the report), and +// B. N notes → one activity came back as ONE, the others deleted (found here, +// and worse: A duplicates, B destroys). +// +// Both were measured end to end on a real 11.13.0 project with mxbuild at 0 +// errors on every side, which is why neither had ever been noticed. +// +// A third loss rides along: an Annotation's own position and size were never +// emitted and were re-invented by the writer, so every round trip moved and +// resized every note. + +// mfWithNotes builds the smallest microflow that can carry notes: two log +// activities between a start and an end. +func mfWithNotes(flows []*microflows.AnnotationFlow, notes ...*microflows.Annotation) *microflows.Microflow { + objs := []microflows.MicroflowObject{ + µflows.StartEvent{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: "start"}, Position: model.Point{X: 0, Y: 100}}}, + µflows.ActionActivity{BaseActivity: microflows.BaseActivity{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: "a1"}, Position: model.Point{X: 100, Y: 100}}}, + Action: µflows.LogMessageAction{LogNodeName: "N", LogLevel: "Info", MessageTemplate: &model.Text{}}}, + µflows.ActionActivity{BaseActivity: microflows.BaseActivity{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: "a2"}, Position: model.Point{X: 200, Y: 100}}}, + Action: µflows.LogMessageAction{LogNodeName: "N", LogLevel: "Info", MessageTemplate: &model.Text{}}}, + µflows.EndEvent{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: "end"}, Position: model.Point{X: 300, Y: 100}}}, + } + for _, n := range notes { + objs = append(objs, n) + } + return µflows.Microflow{ObjectCollection: µflows.MicroflowObjectCollection{ + Objects: objs, + Flows: []*microflows.SequenceFlow{ + {OriginID: "start", DestinationID: "a1"}, + {OriginID: "a1", DestinationID: "a2"}, + {OriginID: "a2", DestinationID: "end"}, + }, + AnnotationFlows: flows, + }} +} + +func note(id, caption string, pos model.Point, size model.Size) *microflows.Annotation { + return µflows.Annotation{ + BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(id)}, Position: pos, Size: size}, + Caption: caption, + } +} + +func describeBody(t *testing.T, mf *microflows.Microflow) string { + t.Helper() + return strings.Join(formatMicroflowActivities(&ExecContext{}, mf, nil, nil), "\n") +} + +// rebuildFromBody parses a described body and returns the notes the flow builder +// produced. The whole point is that the text is the contract: a helper that +// handed the AST straight to the builder would not exercise the describer. +func rebuildFromBody(t *testing.T, body string) (notes []*microflows.Annotation, flows []*microflows.AnnotationFlow, errs []string) { + t.Helper() + src := "create microflow M.F ()\nbegin\n" + body + "\nend;" + prog, parseErrs := visitor.Build(src) + if len(parseErrs) > 0 { + t.Fatalf("DESCRIBE output does not parse: %v\n%s", parseErrs, src) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, + varTypes: map[string]string{}, declaredVars: map[string]string{}} + fb.buildFlowGraph(mf.Body, nil) + for _, o := range fb.objects { + if a, ok := o.(*microflows.Annotation); ok { + notes = append(notes, a) + } + } + return notes, fb.annotationFlows, fb.GetErrors() +} + +// --------------------------------------------------------------------------- +// A — the reported defect +// --------------------------------------------------------------------------- + +// One note wired to two activities must survive as ONE note with two flows. +// +// Control: revert getAnnotationsByTarget to filing bare captions (drop the +// Shared flag) and the rebuild yields 2 notes — the reported symptom. +func TestSharedNote_SurvivesTheRoundTripAsOneNote(t *testing.T) { + mf := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "n", DestinationID: "a1"}, + {BaseElement: model.BaseElement{ID: "af2"}, OriginID: "n", DestinationID: "a2"}, + }, note("n", "shared note", model.Point{X: 175, Y: -40}, model.Size{Width: 260, Height: 70})) + + body := describeBody(t, mf) + if strings.Count(body, "@annotation") != 2 { + t.Fatalf("both activities should still be annotated:\n%s", body) + } + if !strings.Contains(body, "@annotation(id: n1, text: 'shared note', position: (175, -40), size: (260, 70))") { + t.Errorf("the first mention must declare the note with its geometry:\n%s", body) + } + if !strings.Contains(body, "@annotation(id: n1)") { + t.Errorf("the second mention must REFERENCE the note, not repeat it:\n%s", body) + } + + notes, flows, errs := rebuildFromBody(t, body) + if len(errs) > 0 { + t.Fatalf("rebuild reported errors: %v", errs) + } + if len(notes) != 1 { + t.Fatalf("got %d notes, want 1 — the note was duplicated (#1077)", len(notes)) + } + if len(flows) != 2 { + t.Fatalf("got %d annotation flows, want 2 — the note lost a connection", len(flows)) + } + if flows[0].OriginID != notes[0].ID || flows[1].OriginID != notes[0].ID { + t.Errorf("both flows must originate from the one note, got %s and %s (note %s)", + flows[0].OriginID, flows[1].OriginID, notes[0].ID) + } + if got := notes[0].Position; got != (model.Point{X: 175, Y: -40}) { + t.Errorf("position = %v, want (175,-40)", got) + } + if got := notes[0].Size; got != (model.Size{Width: 260, Height: 70}) { + t.Errorf("size = %v, want 260x70", got) + } +} + +// The round trip must be a FIXED POINT, not merely better once. Describing the +// rebuilt flow has to give back the same text — otherwise the model drifts a +// little further on every describe → exec cycle, which is how this defect +// stayed invisible in the first place. +func TestSharedNote_DescribeIsAFixedPoint(t *testing.T) { + mf := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "n", DestinationID: "a1"}, + {BaseElement: model.BaseElement{ID: "af2"}, OriginID: "n", DestinationID: "a2"}, + }, note("n", "shared note", model.Point{X: 175, Y: -40}, model.Size{Width: 260, Height: 70})) + + first := describeBody(t, mf) + notes, flows, _ := rebuildFromBody(t, first) + + rebuilt := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: notes[0].ID, DestinationID: "a1"}, + {BaseElement: model.BaseElement{ID: "af2"}, OriginID: notes[0].ID, DestinationID: "a2"}, + }, notes[0]) + if len(flows) != 2 { + t.Fatalf("precondition: want 2 flows, got %d", len(flows)) + } + + if second := describeBody(t, rebuilt); second != first { + t.Errorf("describe is not a fixed point:\n--- first\n%s\n--- second\n%s", first, second) + } +} + +// --------------------------------------------------------------------------- +// B — the defect the report did not mention, which destroys rather than copies +// --------------------------------------------------------------------------- + +// Control: put back `result.AnnotationText = text` (a single slot) in the +// visitor and only 'second note' survives. +func TestTwoNotesOnOneActivity_BothSurvive(t *testing.T) { + mf := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "n1", DestinationID: "a1"}, + {BaseElement: model.BaseElement{ID: "af2"}, OriginID: "n2", DestinationID: "a1"}, + }, + note("n1", "first note", model.Point{X: 100, Y: 0}, DefaultAnnotationSize), + note("n2", "second note", model.Point{X: 100, Y: -60}, DefaultAnnotationSize), + ) + + body := describeBody(t, mf) + notes, flows, errs := rebuildFromBody(t, body) + if len(errs) > 0 { + t.Fatalf("rebuild reported errors: %v", errs) + } + var captions []string + for _, n := range notes { + captions = append(captions, n.Caption) + } + if len(notes) != 2 { + t.Fatalf("got %v, want both notes — one was silently deleted (#1077)", captions) + } + if captions[0] != "first note" || captions[1] != "second note" { + t.Errorf("captions = %v, want [first note second note]", captions) + } + if len(flows) != 2 { + t.Errorf("got %d flows, want 2", len(flows)) + } + // Both notes point at the same activity, and neither landed on top of the + // other: stacking is what makes fixing B safe to ship without geometry. + if notes[0].Position == notes[1].Position { + t.Errorf("both notes are at %v — they overlap on the canvas", notes[0].Position) + } +} + +// --------------------------------------------------------------------------- +// C — geometry, and the reason the everyday form did not get longer +// --------------------------------------------------------------------------- + +// A note at the position and size the writer re-derives keeps the short form. +// This is what stops the fix churning every microflow that has a note in it, +// and it is only sound because both sides go through +// defaultAnnotationGeometry — see the test below. +func TestUnsharedNoteAtTheDefault_KeepsTheShortForm(t *testing.T) { + pos, size := defaultAnnotationGeometry(model.Point{X: 100, Y: 100}, 0) + mf := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "n1", DestinationID: "a1"}, + }, note("n1", "plain note", pos, size)) + + body := describeBody(t, mf) + if !strings.Contains(body, "@annotation 'plain note'") { + t.Errorf("a note at the default geometry must still emit the short form:\n%s", body) + } + if strings.Contains(body, "at:") || strings.Contains(body, "size:") { + t.Errorf("no geometry should be emitted for a default-placed note:\n%s", body) + } +} + +// A note that was moved or resized on the canvas spells its geometry out, and +// the rebuild honours it exactly. +func TestMovedNote_CarriesItsGeometryThroughTheRoundTrip(t *testing.T) { + mf := mfWithNotes([]*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "n1", DestinationID: "a1"}, + }, note("n1", "moved note", model.Point{X: -40, Y: 900}, model.Size{Width: 420, Height: 30})) + + body := describeBody(t, mf) + if !strings.Contains(body, "@annotation(text: 'moved note', position: (-40, 900), size: (420, 30))") { + t.Fatalf("geometry not emitted:\n%s", body) + } + notes, _, errs := rebuildFromBody(t, body) + if len(errs) > 0 { + t.Fatalf("rebuild reported errors: %v", errs) + } + if len(notes) != 1 { + t.Fatalf("got %d notes, want 1", len(notes)) + } + if notes[0].Position != (model.Point{X: -40, Y: 900}) || notes[0].Size != (model.Size{Width: 420, Height: 30}) { + t.Errorf("geometry lost: pos=%v size=%v", notes[0].Position, notes[0].Size) + } +} + +// The describer decides whether to emit `at:`/`size:` by asking what the writer +// would re-derive. If the two ever disagree the omission becomes a silent drift +// — the note creeps further on every round trip — so this pins that there is +// exactly one formula and both sides use it. +// +// Control: change the -100 in defaultAnnotationGeometry and this still passes +// (both sides moved together), while hardcoding the old constant in either side +// alone fails TestUnsharedNoteAtTheDefault_KeepsTheShortForm. +func TestAnnotationGeometryDefaultIsSharedByBothSides(t *testing.T) { + activity := model.Point{X: 640, Y: 320} + for index := 0; index < 3; index++ { + wantPos, wantSize := defaultAnnotationGeometry(activity, index) + + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, + varTypes: map[string]string{}, declaredVars: map[string]string{}} + fb.objects = append(fb.objects, µflows.ActionActivity{ + BaseActivity: microflows.BaseActivity{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: "act"}, Position: activity}}}) + fb.attachAnnotation(ast.MicroflowAnnotation{Text: "x"}, "act", index) + + var got *microflows.Annotation + for _, o := range fb.objects { + if a, ok := o.(*microflows.Annotation); ok { + got = a + } + } + if got == nil { + t.Fatalf("index %d: no annotation created", index) + } + if got.Position != wantPos || got.Size != wantSize { + t.Errorf("index %d: writer placed the note at %v/%v, describer assumes %v/%v", + index, got.Position, got.Size, wantPos, wantSize) + } + } +} + +// --------------------------------------------------------------------------- +// Authoring the long form directly, and the refusals +// --------------------------------------------------------------------------- + +func TestAuthorSharedNote_OneNoteTwoFlows(t *testing.T) { + notes, flows, errs := rebuildFromBody(t, ` + @annotation(id: shared, text: 'watch out') + log info node 'N' 'one'; + @annotation(id: shared) + log info node 'N' 'two'; + return;`) + if len(errs) > 0 { + t.Fatalf("errors: %v", errs) + } + if len(notes) != 1 || len(flows) != 2 { + t.Fatalf("got %d notes / %d flows, want 1 / 2", len(notes), len(flows)) + } + if notes[0].Caption != "watch out" { + t.Errorf("caption = %q", notes[0].Caption) + } +} + +// Control for the test above: WITHOUT the id, the same two lines must produce +// two separate notes. Otherwise the test above would also pass for an +// implementation that deduplicated on text, which would be a different and +// wrong behaviour — two notes with the same words are still two notes. +func TestAuthorTwoNotesWithTheSameText_StayTwoNotes(t *testing.T) { + notes, flows, errs := rebuildFromBody(t, ` + @annotation 'watch out' + log info node 'N' 'one'; + @annotation 'watch out' + log info node 'N' 'two'; + return;`) + if len(errs) > 0 { + t.Fatalf("errors: %v", errs) + } + if len(notes) != 2 || len(flows) != 2 { + t.Fatalf("got %d notes / %d flows, want 2 / 2 — identical text must not be merged", + len(notes), len(flows)) + } +} + +func TestAuthorNote_UndeclaredIDIsRefused(t *testing.T) { + _, _, errs := rebuildFromBody(t, ` + @annotation(id: ghost) + log info node 'N' 'one'; + return;`) + if len(errs) == 0 { + t.Fatal("a reference to a note that was never given text must be refused") + } + if !strings.Contains(strings.Join(errs, "\n"), "ghost") { + t.Errorf("the error should name the id: %v", errs) + } +} + +func TestAuthorNote_SameIDWithDifferentTextIsRefused(t *testing.T) { + _, _, errs := rebuildFromBody(t, ` + @annotation(id: n, text: 'one thing') + log info node 'N' 'one'; + @annotation(id: n, text: 'another thing') + log info node 'N' 'two'; + return;`) + if len(errs) == 0 { + t.Fatal("one id naming two different notes must be refused") + } +} + +// MDL079 catches at `check` time what the builder refuses at exec time. It has +// to be a separate check because a label is declared on one statement and used +// on another, and because the builder's errors do not escape a loop body. +func TestMDL079_RefusesAnUndeclaredNoteID(t *testing.T) { + if ids := validateMicroflowSource(t, ` + @annotation(id: ghost) + log info node 'N' 'one'; + return;`); !reportsRule(ids, "MDL079") { + t.Errorf("check should report MDL079, got %v", ids) + } +} + +// The control: the same script WITH the declaration must be clean, or MDL079 +// would be firing on everything and proving nothing. +func TestMDL079_AcceptsADeclaredNoteID(t *testing.T) { + if ids := validateMicroflowSource(t, ` + @annotation(id: ok, text: 'declared here') + log info node 'N' 'one'; + @annotation(id: ok) + log info node 'N' 'two'; + return;`); reportsRule(ids, "MDL079") { + t.Errorf("a declared id must not be reported: %v", ids) + } +} + +func TestMDL079_RefusesAnUnknownParameter(t *testing.T) { + if ids := validateMicroflowSource(t, ` + @annotation(txt: 'typo in the key') + log info node 'N' 'one'; + return;`); !reportsRule(ids, "MDL079") { + t.Errorf("an unknown @annotation parameter must be reported, got %v", ids) + } +} + +func validateMicroflowSource(t *testing.T, body string) []string { + t.Helper() + prog, errs := visitor.Build("create microflow M.F ()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + var ids []string + for _, v := range ValidateMicroflow(mf) { + ids = append(ids, v.RuleID) + } + return ids +} + +func reportsRule(ids []string, ruleID string) bool { + for _, id := range ids { + if id == ruleID { + return true + } + } + return false +} + +// --------------------------------------------------------------------------- +// Adjacent gaps found while fixing the above +// --------------------------------------------------------------------------- + +// A note written inside an `on error { … }` body was dropped: the handler's +// sub-builder collected annotation flows into its own slice, and the merge back +// into the parent copied objects and sequence flows but not annotation flows. +// The Annotation object arrived, the edge did not — so the note came back as a +// free-floating one, detached from the activity it documented. +func TestNoteInsideAnErrorHandlerKeepsItsConnection(t *testing.T) { + notes, flows, errs := rebuildFromBody(t, ` + $Car = create M.Car () on error { + @annotation 'why this failed' + log error node 'N' 'boom'; + }; + return;`) + if len(errs) > 0 { + t.Fatalf("errors: %v", errs) + } + if len(notes) != 1 { + t.Fatalf("got %d notes, want 1", len(notes)) + } + if len(flows) != 1 { + t.Fatalf("got %d annotation flows, want 1 — the note lost its connection", len(flows)) + } + if flows[0].OriginID != notes[0].ID { + t.Errorf("flow originates from %s, want the note %s", flows[0].OriginID, notes[0].ID) + } +} + +// The read half of the same gap. collectErrorHandlerStatements is a second, +// smaller describer — the main traversal never steps inside an `on error { … }` +// block — and it emitted no annotations at all. Fixing only the write half +// would have left the round trip losing the note exactly as before, with the +// model now merely differently wrong. +func TestNoteInsideAnErrorHandlerIsDescribed(t *testing.T) { + notes, flows, errs := rebuildFromBody(t, ` + $Car = create M.Car () on error { + @annotation 'why this failed' + log error node 'N' 'boom'; + }; + return;`) + if len(errs) > 0 || len(notes) != 1 || len(flows) != 1 { + t.Fatalf("precondition: notes=%d flows=%d errs=%v", len(notes), len(flows), errs) + } + + // Rebuild the described microflow from the model the builder produced, and + // describe THAT — the note has to come back out of the handler body. + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, + varTypes: map[string]string{}, declaredVars: map[string]string{}} + prog, _ := visitor.Build(`create microflow M.F () +begin + $Car = create M.Car () on error { + @annotation 'why this failed' + log error node 'N' 'boom'; + }; + return; +end;`) + oc := fb.buildFlowGraph(prog.Statements[0].(*ast.CreateMicroflowStmt).Body, nil) + body := describeBody(t, µflows.Microflow{ObjectCollection: oc}) + if !strings.Contains(body, "why this failed") { + t.Errorf("the handler-body note was dropped by DESCRIBE:\n%s", body) + } +} + +// A refusal raised inside a loop body used to be swallowed: the loop's +// sub-builder collected errors into its own slice and nothing merged them back, +// so exec reported success and wrote a flow with the note missing. `check` +// (MDL079) catches this shape too, but the two must agree. +func TestNoteRefusalInsideALoopBodyIsNotSwallowed(t *testing.T) { + _, _, errs := rebuildFromBody(t, ` + declare $Items list of M.Car = empty; + loop $Item in $Items begin + @annotation(id: ghost) + log info node 'N' 'inside'; + end loop; + return;`) + if len(errs) == 0 { + t.Fatal("a bad note reference inside a loop body must reach the caller") + } +} + +// A note declared outside a loop and attached to an activity inside it. The +// describer emits this shape (its label state spans the loop overlay), so the +// builder has to accept it or exec would refuse mxcli's own DESCRIBE output. +// +// Measured at 0 errors on mxbuild 11.13 before this was allowed — see the +// comment on the loop sub-builder. +func TestNoteSharedAcrossALoopBoundary(t *testing.T) { + notes, flows, errs := rebuildFromBody(t, ` + @annotation(id: n, text: 'spans the loop') + log info node 'N' 'outer'; + loop $Item in $Items begin + @annotation(id: n) + log info node 'N' 'inner'; + end loop; + return;`) + if len(errs) > 0 { + t.Fatalf("errors: %v", errs) + } + if len(notes) != 1 { + t.Fatalf("got %d notes, want 1", len(notes)) + } + if len(flows) != 2 { + t.Fatalf("got %d annotation flows, want 2", len(flows)) + } +} + +// `check` and `exec` must refuse the SAME scripts. A reference written above +// its declaration is one the builder cannot satisfy — it walks statements in +// order — so MDL079 refuses it too rather than passing a script exec will then +// reject. DESCRIBE never emits this shape. +func TestMDL079_AgreesWithTheBuilderOnDeclarationOrder(t *testing.T) { + body := ` + @annotation(id: n) + log info node 'N' 'one'; + @annotation(id: n, text: 'declared too late') + log info node 'N' 'two'; + return;` + + if ids := validateMicroflowSource(t, body); !reportsRule(ids, "MDL079") { + t.Errorf("check accepted a reference above its declaration, got %v", ids) + } + if _, _, errs := rebuildFromBody(t, body); len(errs) == 0 { + t.Error("exec accepted a reference above its declaration") + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index cb00ce949a..ef0bf91e4b 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -101,6 +101,7 @@ func (v *microflowValidator) addViolation(ruleID string, severity linter.Severit // validate runs all checks on the microflow body. func (v *microflowValidator) validate(body []ast.MicroflowStatement) { v.checkListOperationIterator(body) + v.checkAnnotationLabels(body) // Walk the body for per-statement checks (validation feedback, return value checks) v.emptyListVars = make(map[string]bool) @@ -1518,4 +1519,53 @@ func (v *microflowValidator) checkUnknownAnnotations(s ast.MicroflowStatement) { "@excluded, @anchor, @curve and @merge on a microflow statement. If `@%s` is a typo of "+ "one of those, correct it; container size is not authorable (upstream #884).", name)) } + for _, bad := range ann.InvalidNotes { + v.addViolation("MDL079", linter.SeverityError, + fmt.Sprintf("`@annotation` parameter `%s` is not one mxcli understands, so the note it "+ + "belongs to is not written at all", bad), + "A note is either `@annotation 'text'`, or the long form "+ + "`@annotation(id: n1, text: 'text', position: (x, y), size: (w, h))` where every parameter "+ + "except one of `id:`/`text:` is optional. `id:` names a note so a later "+ + "`@annotation(id: n1)` attaches THAT note to another activity instead of creating a "+ + "second one (mendixlabs/mxcli#1077).") + } +} + +// checkAnnotationLabels refuses an `@annotation(id: …)` that never gets a text. +// +// This is a whole-body check rather than a per-statement one because a label is +// declared on one statement and referenced on another — the point of having +// labels at all. The flow builder refuses the same thing at exec time, but its +// errors do not escape a loop body, and `check` is what people run. +func (v *microflowValidator) checkAnnotationLabels(body []ast.MicroflowStatement) { + // One pass, in statement order, because that is what the flow builder does: + // a two-pass check would accept a reference written above its declaration + // and exec would then refuse it. DESCRIBE never emits that shape either — it + // declares a note at its first mention in traversal order — so agreeing with + // the builder costs nothing and keeps `check` honest. + declared := map[string]bool{} + var walk func([]ast.MicroflowStatement) + walk = func(stmts []ast.MicroflowStatement) { + for _, s := range stmts { + if ann := ast.StatementAnnotations(s); ann != nil { + for _, note := range append(append([]ast.MicroflowAnnotation{}, ann.Notes...), ann.FreeNotes...) { + switch { + case note.Label == "": + case note.Text != "": + declared[note.Label] = true + case !declared[note.Label]: + v.addViolation("MDL079", linter.SeverityError, + fmt.Sprintf("`@annotation(id: %s)` refers to a note that has not been declared above it", note.Label), + fmt.Sprintf("The FIRST mention of a note carries its text: "+ + "`@annotation(id: %s, text: '…')`. Later mentions attach that same note to "+ + "another activity with `@annotation(id: %s)`.", note.Label, note.Label)) + } + } + } + for _, nested := range ast.StatementBodies(s) { + walk(nested) + } + } + } + walk(body) } diff --git a/mdl/executor/validate_microflow_loop_caption_test.go b/mdl/executor/validate_microflow_loop_caption_test.go index 2776950396..d66e27e2aa 100644 --- a/mdl/executor/validate_microflow_loop_caption_test.go +++ b/mdl/executor/validate_microflow_loop_caption_test.go @@ -41,7 +41,7 @@ func TestValidateMicroflow_CaptionOnLoopWarns(t *testing.T) { // @annotation (the supported way to label a loop) must NOT warn. func TestValidateMicroflow_AnnotationOnLoopNoWarn(t *testing.T) { - if loopHasMDL042(mfWithLoopAnnotations(&ast.ActivityAnnotations{AnnotationText: "Process things"})) { + if loopHasMDL042(mfWithLoopAnnotations(&ast.ActivityAnnotations{Notes: []ast.MicroflowAnnotation{{Text: "Process things"}}})) { t.Error("MDL042 must not fire for @annotation on a loop") } } diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index a27ba981a7..70afbc53e7 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -508,6 +508,14 @@ annotationParamName | TRUE | FALSE | TAIL // @anchor(... tail: (...)) + // @annotation(id: n1, text: '…', position: (x, y), size: (w, h)) — #1077. + // `id` and `size` are already IDENTIFIER; these two are lexer keywords, and + // a keyword key does NOT fail to parse — annotationParam falls through to + // its positional alternative, so the parameter is accepted and silently + // means nothing. Anything added here must be listed, not assumed. + | POSITION // the note's own place on the canvas, distinct from the + // @position of the activity it documents + | TEXT ; annotationValue diff --git a/mdl/visitor/visitor_microflow_statements.go b/mdl/visitor/visitor_microflow_statements.go index 0acaa20149..ea98db5b1d 100644 --- a/mdl/visitor/visitor_microflow_statements.go +++ b/mdl/visitor/visitor_microflow_statements.go @@ -284,17 +284,30 @@ func extractMicroflowAnnotations(annotations []parser.IAnnotationContext) *ast.A seenActivityMetadata = true case "annotation": - // @annotation 'text' — bare annotationValue - if valCtx := ann.AnnotationValue(); valCtx != nil { - text := extractAnnotationValueString(valCtx) - if text != "" { - if !seenActivityMetadata && hasLaterActivityAnnotation(annotations, i+1) { - result.FreeAnnotations = append(result.FreeAnnotations, text) - } else { - result.AnnotationText = text - } - hasAny = true + // Two forms. `@annotation 'text'` is the everyday one and is + // unchanged. `@annotation(id: n1, text: '…', position: (x, y), + // size: (w, h))` carries the note's identity and geometry, which the + // bare form cannot express (#1077). + // + // The annotation rule already accepted parenthesised params, so that + // form PARSED before this and was silently discarded. Two of the + // four keys still needed a grammar change: `text` and `position` are + // lexer keywords, and a keyword key does not fail the parse — it + // falls through to annotationParam's positional alternative — so + // they were being accepted and quietly ignored. + note, ok := parseNoteAnnotation(ann, result) + if ok { + // Free-floating only when nothing has claimed this statement + // yet AND an activity annotation follows: the note belongs to + // the canvas, not to the statement below it. + if !seenActivityMetadata && hasLaterActivityAnnotation(annotations, i+1) { + result.FreeNotes = append(result.FreeNotes, note) + } else { + result.Notes = append(result.Notes, note) } + hasAny = true + } else if len(result.InvalidNotes) > 0 { + hasAny = true } case "excluded": @@ -373,6 +386,79 @@ func extractMicroflowAnnotations(annotations []parser.IAnnotationContext) *ast.A return result } +// parseNoteAnnotation reads one `@annotation` into a MicroflowAnnotation. +// +// Anything it cannot use is recorded on result.InvalidNotes rather than +// dropped, and makes the note itself invalid: a typo'd parameter would +// otherwise cost the reader the whole note, or — worse for `id:` — turn a +// reference to an existing note into a second, textless one. Validation refuses +// them (MDL079); the visitor's job is only to not lose them. +func parseNoteAnnotation(ann *parser.AnnotationContext, result *ast.ActivityAnnotations) (ast.MicroflowAnnotation, bool) { + var note ast.MicroflowAnnotation + + // @annotation 'text' — the bare form. + if valCtx := ann.AnnotationValue(); valCtx != nil { + note.Text = extractAnnotationValueString(valCtx) + return note, note.Text != "" + } + + params := ann.AnnotationParams() + if params == nil { + return note, false + } + + for _, p := range params.(*parser.AnnotationParamsContext).AllAnnotationParam() { + pCtx := p.(*parser.AnnotationParamContext) + nameCtx := pCtx.AnnotationParamName() + if nameCtx == nil { + // Positional. Deliberately unsupported: `@annotation('a', 'b')` + // has no reading that is obviously right, and guessing one would + // silently mean something. + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + continue + } + switch strings.ToLower(nameCtx.GetText()) { + case "id": + if v := pCtx.AnnotationValue(); v != nil { + note.Label = extractAnnotationValueIdentifier(v) + } + if note.Label == "" { + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + } + case "text": + if v := pCtx.AnnotationValue(); v != nil { + note.Text = extractAnnotationValueString(v) + } + if note.Text == "" { + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + } + case "position": + pt, ok := annotationPointValue(pCtx) + if !ok { + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + continue + } + note.Position = pt + case "size": + pt, ok := annotationPointValue(pCtx) + if !ok { + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + continue + } + note.Size = &ast.BoxSize{Width: pt.X, Height: pt.Y} + default: + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(pCtx.GetText())) + } + } + + // A note needs either text (a declaration) or a label (a reference to one). + if note.Text == "" && note.Label == "" { + result.InvalidNotes = append(result.InvalidNotes, strings.TrimSpace(ann.GetText())) + return note, false + } + return note, true +} + func hasLaterActivityAnnotation(annotations []parser.IAnnotationContext, start int) bool { for _, annCtx := range annotations[start:] { ann := annCtx.(*parser.AnnotationContext) diff --git a/mdl/visitor/visitor_test.go b/mdl/visitor/visitor_test.go index d2dd046759..9ffbcaab76 100644 --- a/mdl/visitor/visitor_test.go +++ b/mdl/visitor/visitor_test.go @@ -1962,11 +1962,11 @@ END;` if logStmt.Annotations == nil { t.Fatal("expected annotations") } - if got := logStmt.Annotations.FreeAnnotations; len(got) != 1 || got[0] != "free note" { + if got := logStmt.Annotations.FreeNotes; len(got) != 1 || got[0].Text != "free note" { t.Fatalf("free annotations = %#v, want [free note]", got) } - if logStmt.Annotations.AnnotationText != "" { - t.Fatalf("attached annotation = %q, want empty", logStmt.Annotations.AnnotationText) + if len(logStmt.Annotations.Notes) != 0 { + t.Fatalf("attached annotations = %#v, want empty", logStmt.Annotations.Notes) } } @@ -1996,11 +1996,12 @@ END;` t.Fatal("expected annotations") } want := []string{"first free note", "second free note", "third free note"} - if got := logStmt.Annotations.FreeAnnotations; len(got) != len(want) || got[0] != want[0] || got[1] != want[1] || got[2] != want[2] { + got := logStmt.Annotations.FreeNotes + if len(got) != len(want) || got[0].Text != want[0] || got[1].Text != want[1] || got[2].Text != want[2] { t.Fatalf("free annotations = %#v, want %#v", got, want) } - if logStmt.Annotations.AnnotationText != "" { - t.Fatalf("attached annotation = %q, want empty", logStmt.Annotations.AnnotationText) + if len(logStmt.Annotations.Notes) != 0 { + t.Fatalf("attached annotations = %#v, want empty", logStmt.Annotations.Notes) } } @@ -2027,11 +2028,11 @@ END;` if logStmt.Annotations == nil { t.Fatal("expected annotations") } - if logStmt.Annotations.AnnotationText != "attached note" { - t.Fatalf("attached annotation = %q, want attached note", logStmt.Annotations.AnnotationText) + if got := logStmt.Annotations.Notes; len(got) != 1 || got[0].Text != "attached note" { + t.Fatalf("attached annotations = %#v, want [attached note]", got) } - if len(logStmt.Annotations.FreeAnnotations) != 0 { - t.Fatalf("free annotations = %#v, want empty", logStmt.Annotations.FreeAnnotations) + if len(logStmt.Annotations.FreeNotes) != 0 { + t.Fatalf("free annotations = %#v, want empty", logStmt.Annotations.FreeNotes) } }