From 60123e24c855847ba78704d84e507c7833d06770 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 18:19:30 +0000 Subject: [PATCH 01/18] docs(soap): measure the proposed syntax against the sibling call statements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MDL already has three other ways to call a remote operation, and the proposal asserted consistency with them without showing the comparison. Shown now, with the one disagreement named: - The argument form is `call external action`'s exactly — an OData action and a SOAP operation are the same shape of thing, and MDL already spells one `Action(name = value)`, parenthesised on the callee. That is also why the args belong on `operation` rather than in a clause: SOAP's statement target is the service, so `operation X` occupies the position the OData action's qualified name does. - `send mapping … from $var` is `rest call`'s `body mapping … from $var`, keyword for keyword after the noun. - `send rest request` diverges, and is itself the outlier: it is the only place in MDL where a parameter name carries a `$`. callArgumentList accepts both spellings, so nothing forces it, and `call microflow Mod.M (FirstName = 'Hello')` is what the rest of the language does. Matching it would spread the wart; a separate issue, out of scope here. Also records a precedent for §2.1's derivation found in that same statement: RestOperationCallAction stores each parameter under a qualified key and the describer strips everything before the last dot, so MDL shows `code` where the model holds `Mod.Svc.Op.code`. Deriving ParameterPath from the operation's RequestBodyElementName is the same move with a different separator. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../PROPOSAL_soap_request_body.md | 52 +++++++++++++++++-- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_soap_request_body.md b/docs/11-proposals/PROPOSAL_soap_request_body.md index 4368f29a5..b9f9467e7 100644 --- a/docs/11-proposals/PROPOSAL_soap_request_body.md +++ b/docs/11-proposals/PROPOSAL_soap_request_body.md @@ -212,13 +212,55 @@ callWebServiceStatement `FROM` and `LPAREN`/`RPAREN` are existing tokens; no lexer change. +### 3.1 Measured against the sibling call statements + +MDL already has three other ways to call a remote operation. Both halves of the +proposal are taken from them rather than invented, and the one statement they +disagree with is the one that is already out of step with the rest of the +language. + +| statement | arguments | request body from a mapping | +|---|---|---| +| `call external action` (OData action) | `Ext.TripPin.FindAirport(code = 'EHAM')` — bare names, parenthesised **on the callee** | n/a | +| `rest call` (REST v1, inline HTTP) | `with ({1} = $x)` — positional **URL template slots**, a different concept | `body mapping Mod.M from $var` | +| `send rest request` (REST v2, consumed service operation) | `with ($code = 'EHAM')` — `$`-prefixed, in a separate clause | `body $var` (the mapping lives on the operation document) | +| `call microflow` / `nanoflow` / `java action` / `execute database query` | `(Name = value)` via `callArgumentList` | n/a | +| **SOAP, proposed** | `operation GetOrder (OrderId = $Id)` | `send mapping Mod.M from $var` | + +- **The argument form is the OData action's, exactly.** An OData action and a + SOAP operation are the same shape of thing — a named operation on a consumed + service, with declared parameters — and MDL already spells one + `Action(name = value)`. Parenthesising on the callee is also why the args + belong on `operation` rather than in a clause of their own: SOAP's statement + target is the *service*, so `operation X` is the callee, in the position + `call external action`'s qualified name occupies. +- **`send mapping … from $var` is `rest call`'s `body mapping … from $var`**, + keyword for keyword after the noun. `send`/`body` differ only because `send + mapping` is already in the SOAP grammar; renaming it would break scripts for + nothing. +- **`send rest request` is the divergence, and it is the outlier.** It is the + only place in MDL where a *parameter* name carries a `$` — DESCRIBE emits + `with ($code = …)` — which reads as an assignment to a variable rather than a + binding of a parameter. `callArgumentList` accepts both spellings, so nothing + forces the `$`; `call microflow Mod.M (FirstName = 'Hello')` is what the rest + of the language does. Matching it here would spread the wart. Worth a separate + issue on its own; out of scope for this proposal. + +There is also a **precedent for §2.1's derivation**, in that same statement: +`RestOperationCallAction` stores each parameter under a qualified key and the +describer strips everything before the last dot (`formatRestOperationCallAction`), +so MDL shows `code` where the model holds `Mod.Svc.Op.code`. Deriving +`ParameterPath` from the operation's `RequestBodyElementName` is the same move +with a different separator — which makes it an established pattern here rather +than a new liberty. + Against the design checklist: it reads as English (*call this service, operation GetOrder with OrderId …*); it adds no verb; both clauses are optional, so every script that parses today still parses; one argument is a one-line diff; and an LLM that has seen `call microflow Mod.Flow (Name = $x)` generates this correctly from the shape alone. -### 3.1 The clauses are mutually exclusive — and that is the rule to enforce +### 3.2 The clauses are mutually exclusive — and that is the rule to enforce `RequestBodyHandling` holds one variant. So: @@ -237,7 +279,7 @@ should name both clauses and say that a call sends either arguments or a mapping Per the repo's rule, `check` and `exec` must call the **same** function, so a script cannot pass one and fail the other. -### 3.2 DESCRIBE +### 3.3 DESCRIBE Round-trippable, per the layouts precedent — describe → edit → exec is how a SOAP call gets copied. Once `RequestBodyHandling` is representable, remove it from the @@ -262,7 +304,7 @@ reconsidering only as an escape hatch if a real WSDL turns up whose paths are no `element|parameter` — none of TestApp's three are, but three is not many. **Write the send mapping now and leave arguments for later.** Tempting, since the -mapping is four keys. Rejected because the mutual exclusivity in §3.1 is only +mapping is four keys. Rejected because the mutual exclusivity in §3.2 is only enforceable once both exist; implementing one alone means `check` can refuse a combination it cannot yet offer an alternative to. @@ -291,7 +333,7 @@ Roughly the shape the CE0386/CE0243 fixes took, and mostly reusing their parts. path currently looks for `RequestHandling` → `ExportMappingCall` → `Mapping`, a key **no TestApp document carries**, which is why `SendMappingID` was never populated from a real project either. -6. **Validation**: §3.1, one function, called by `check` and `exec`. +6. **Validation**: §3.2, one function, called by `check` and `exec`. 7. **Version gating**: none expected — SOAP calls predate the supported range — but confirm against `sdk/versions/mendix-{9,10,11}.yaml` before merging. @@ -335,7 +377,7 @@ a control.** `http%3A//www.example.com/:GetOrder|OrderId` character for character, since a plausible-looking wrong escaping is exactly what mxbuild would accept and Studio Pro would not. -- Unit: the §3.1 refusal, and that `check` and `exec` reject the same script. +- Unit: the §3.2 refusal, and that `check` and `exec` reject the same script. - `mdl-examples/doctype-tests/06b-soap-examples.mdl` extended with both forms. - **Integration, against ako/TestApp**: rewrite `Clients.GetOrders` and `Clients.SaveOrder` from MDL and get **0 errors** from `mx check`. The control From 7261dfdd1ac2114a9c0c6f6f3e821aa1c25c742e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 19:17:15 +0000 Subject: [PATCH 02/18] =?UTF-8?q?feat(soap):=20make=20the=20request=20body?= =?UTF-8?q?=20authorable=20=E2=80=94=20arguments=20and=20send=20mapping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CALL WEB SERVICE could say who to call and what to do with the answer, and not what to send. Measured on ako/TestApp (11.14.0, baseline 0 errors), the same two-statement script goes from [CE0178] "Body parameter mapping needs to be refreshed." [CE0369] "Cannot use simple request body, as the operation's body is complex" to 0 errors, on both engines. The control is that script with the two new clauses removed, which still gives exactly those two. Implements docs/11-proposals/PROPOSAL_soap_request_body.md. ## One property, not two gaps RequestBodyHandling is a polymorphic child holding EITHER the operation's arguments (SimpleRequestHandling + WebServiceOperationSimpleParameterMapping) or an export mapping (MappingRequestHandling). Both writers emitted an unconditional empty Simple form, so an operation taking parameters had none and `send mapping` — which parsed and was accepted — appeared zero times in the written document. That they are one property is why they land together: the mutual-exclusion rule is only enforceable once both exist. MDL-SOAP01 refuses a statement asking for each, and `check` and `exec` call the same function. $Order = call web service Clients.OrderSoapClient operation GetOrder (OrderId = $Customer/OrderId) receive mapping Clients.SoapOrdersImportMapping; call web service Clients.OrderSoapClient operation SaveOrder send mapping Clients.SoapOrderExportMapping from $NewSaveOrder; Arguments reuse callArgumentList — the (Name = value) form five other call statements already use — and parenthesise on OPERATION, matching CALL EXTERNAL ACTION. `from $var` mirrors REST's `body mapping … from $var`. ## What made readable syntax possible The stored key is a ParameterPath — http%3A//www.example.com/:GetOrder|OrderId — and it is derivable: escape(operation.RequestBodyElementName) + "|" + name, with the element read off Description.Services[].Operations[] of the imported service. So a script names only the parameter. Escaping is per SEGMENT (a ':' inside one becomes %3A; the separator ':' and the slashes are left alone), and a segment containing '%' is refused rather than guessed at. An operation that cannot be resolved is refused too, unlike the other resolvers here, which fall back. Falling back is safe when the alternative is the value that ships today; there is no shipping value for a path never written, and a fabricated one reproduces CE0178 with different text in it. ## Three traps, each with a control - A populated ParameterMappings list leads with typed-array marker 2 and the codec DEFAULTS TO 3. Nothing in a build catches that; Studio Pro refuses to open the project. Stubbing the RegisterListMarker makes the test report "marker = 3, want int32(2)". - MappingRequestHandling's two name keys are MappingId and MappingVariableName. gen binds them as Mapping and MappingArgumentVariableName — both already in its key audit — so they are written raw on the codec side. - ContentType is "Json" on the one reference send mapping, on an XML protocol. Written as observed, carried through a rewrite rather than normalised, and flagged as needing a second reference. ## DESCRIBE, and the regression a round trip caught The structured describe form was unreachable for every real call: they carry fifteen keys and only nine were admitted, so Studio Pro's and mxcli's alike rendered as base64. Admitting the other six BY NAME turned out to be wrong — a describe → exec over the reference documents flipped Range.SingleObject false → true with no error at all, and turned SaveOrder's DataTypes$BooleanType result (the WSDL operation's own return type) into VoidType, i.e. CE0366 + CE6011. So the gate now asks whether writing the document back would reproduce it, not whether the key is known: each boilerplate key is admitted only AT the value mxcli writes. mxcli-authored calls describe structurally and survive a forced round trip byte-identical; Studio Pro's keep the raw fallback until Range.SingleObject is explained. Both engines implement this separately, so a paired test holds them together. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .../fix-issue/findings/mdl-executor.jsonl | 1 + .../write-microflows/reference/integration.md | 54 +++- cmd/mxcli/syntax/features_integration.go | 55 ++++ docs/01-project/MDL_QUICK_REFERENCE.md | 7 + .../doctype-tests/06b-soap-examples.mdl | 32 +- mdl/ast/ast_microflow.go | 23 +- .../modelsdk/microflow_read_actions.go | 260 ++++++++++++++++- .../modelsdk/microflow_webservice_write.go | 116 ++++++-- .../microflow_webservice_write_test.go | 258 ++++++++++++++++ mdl/executor/cmd_microflows_builder_calls.go | 66 ++++- mdl/executor/cmd_microflows_format_action.go | 36 ++- .../cmd_microflows_format_action_test.go | 64 ++++ mdl/executor/validate_microflow.go | 4 + .../validate_webservice_request_body.go | 87 ++++++ .../validate_webservice_request_body_test.go | 100 +++++++ mdl/executor/webservice_names.go | 104 ++++++- mdl/executor/webservice_names_test.go | 92 ++++++ mdl/grammar/domains/MDLMicroflow.g4 | 16 +- mdl/visitor/visitor_microflow_actions.go | 17 +- mdl/visitor/visitor_webservice_test.go | 61 ++++ sdk/microflows/microflows_actions.go | 48 ++- sdk/mpr/parser_microflow_actions.go | 196 ++++++++++++- sdk/mpr/writer_microflow_actions.go | 77 ++++- sdk/mpr/writer_webservice_body_test.go | 275 ++++++++++++++++++ 25 files changed, 1970 insertions(+), 80 deletions(-) create mode 100644 mdl/executor/validate_webservice_request_body.go create mode 100644 mdl/executor/validate_webservice_request_body_test.go create mode 100644 sdk/mpr/writer_webservice_body_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index fee1d0182..adb37ab05 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -89,3 +89,4 @@ {"area": "mdl/backend", "date": "2026-09-09", "symptom": "`call web service` (legacy SOAP) on the DEFAULT engine: `mxcli exec 06b-soap-examples.mdl` reported success on all three microflows and `mx check` (11.13.0) then failed the project with `[CE0008] \"No action defined.\" at Action activity 'Activity'` plus two `[CE0109] \"Undefined variable 'Root'.\"`. Only reachable by rerunning with MXCLI_ENGINE=legacy, which is why legacy was still the documented fallback", "cause": "The codec engine READ the action (`actionFromGen` \u2192 `*microflows.WebServiceCallAction`, with a raw fallback) but `microflowActionToGen` had no case for it, so it hit `default: return nil` and the enclosing ActionActivity serialized with no action at all. The #850 shape: a missing WRITE case is not an unsupported feature, it is a silent drop that exec reports as success. The CE0109s are knock-on \u2014 the dropped action never bound $Root", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go` (new), `microflow_write.go` (switch case)", "insight": "**Mirror the legacy serializer, and prove it by diffing the two engines' documents \u2014 do not re-derive the shape from the metamodel.** There is no Studio Pro-authored SOAP document in this repo, so legacy's output is the only reference that exists and is also what users' projects already contain. Method: exec the same script on each engine, `mxcli bson dump` both, normalise the random $IDs, diff. Three discrepancies fell out that no amount of reading would have shown, all in how the codec emits NULLS and MARKERS: (1) a Part property with no child encodes to nil and the encoder then SKIPS the key (`if val != nil`), so an unset part is an ABSENT key, never a null one \u2014 carry the null as a primitive `bson.Null{}` value instead, which marshals in place; (2) `codec.TypeDefaults.NullFields` does emit the key but APPENDS it after every property, so it cannot reproduce alphabetical key order; (3) both NullFields and list markers are registered per `$Type` and several types are SHARED between writers \u2014 `Microflows$HttpConfiguration` needs HttpHeaderEntries marker 3 for SOAP and 2 for REST, and legacy writes CustomLocationTemplate as null for SOAP but omits it for REST, so a global registration would have silently changed the REST path. Write those explicitly per call site. (The package already carries one such collision: `Microflows$HttpHeaderEntry` is registered 2 in microflow_write.go and 3 in odata_write.go, decided by file order.) Verified: the two engines' CallWebServiceAction documents are now identical key-for-key and value-for-value; `mx check` goes 3 structural errors \u2192 0, leaving only the script's 4 deliberate CE1613 dangling refs, which is exactly legacy's output. Revert control: removing the switch case fails all six unit tests with the CE0008 message. The doctype engineScriptSkip for modelsdk/06b was removed and the script now runs on BOTH engines", "refs": []} {"area": "mdl-backend", "date": "2026-09-10", "symptom": "`mxcli diff-local` on an MPR v2 project fails with `Error: mprcontents directory not found` while mprcontents/ exists and is populated; `MXCLI_ENGINE=legacy` works", "cause": "The modelsdk engine (the default) never overrode `Backend.ContentsDir()`, so it fell through to the generated `unimplemented` stub and returned \"\". diff-local reads \"\" as 'not a v2 project'.", "file": "mdl/backend/modelsdk/backend.go", "insight": "gen_unimplemented.go's promise that an unoverridden method 'fails loudly rather than silently dropping data' is CONDITIONAL on the method having an error to fail through: the generated body is `errUnimplemented` only when a result is `error`, a panic when there are no results at all, and a silent `var r0 T; return r0` otherwise. ContentsDir is in the third bucket and its zero value is a MEANINGFUL in-band answer (\"\" == MPR v1), so the missing implementation was indistinguishable from a v1 project rather than looking like a bug. The detectable signature was the contradiction between two questions the same command asks: Version() (implemented) says 2, ContentsDir() says v1. Guard added in mdl/backend/modelsdk/unimplemented_silent_test.go \u2014 reflect over FullBackend for error-less methods, go/parser the package for methods actually declared on *Backend, since reflection cannot tell a promoted method from an override (Go synthesises a wrapper named (*Backend).X for both). It immediately found a second one, InvalidateCache (a latent panic, no caller today).", "refs": ["mendixlabs/mxcli#1080"]} {"area": "mdl/backend", "date": "2026-09-10", "symptom": "SOAP `call web service` writes a document mxbuild accepts and Studio Pro would not have written. A SEND MAPPING is silently DROPPED by both engines \u2014 `send mapping Mod.Export` parses, `mxcli check` passes, `exec` reports success, and nothing in the stored action references the mapping. Operation ARGUMENTS are dropped the same way", "cause": "sdk/mpr.serializeWebServiceCallAction was written without a Studio Pro reference and hardcodes five things it cannot know, and the codec engine's new writer reproduced it deliberately for parity. Measured against three Studio Pro-authored calls in ako/TestApp (Mendix 11.14.0, Clients.GetOrders / GetCustomerOrders / SaveOrder): ServiceName is the WSDL SERVICE name (\"OrdersWS\") not the local part of the imported service's qualified name (\"OrderSoapClient\"); ImportMappingCall.ContentType is \"Xml\" for a SOAP import mapping, not \"Json\"; Range.SingleObject follows cardinality (false for a list) rather than being always true; VariableType is the real result type (DataTypes$ObjectType with an Entity, DataTypes$BooleanType) rather than always DataTypes$VoidType; and a send mapping is Microflows$MappingRequestHandling {ContentType, MappingId, MappingVariableName}. Arguments live in RequestBodyHandling.ParameterMappings as Microflows$WebServiceOperationSimpleParameterMapping entries keyed by an escaped ParameterPath (\"http%3A//www.example.com/:GetOrder|OrderId\")", "file": "`sdk/mpr/writer_microflow_actions.go` (serializeWebServiceCallAction), `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**A guessed type name in a comment becomes a permanent refusal.** Legacy refused send mappings citing `Mendix$AdvancedRequestHandling`, said it 'requires a Studio Pro-generated example to determine the correct type storage name', and that refusal then shipped for as long as nobody went looking. The real type is `Microflows$MappingRequestHandling` \u2014 which THIS CODEBASE ALREADY WRITES for REST result/request handling \u2014 and the guessed name occurs in none of the three reference documents. The lesson is not about SOAP: when a writer refuses because a storage name is unknown, check whether a sibling feature already writes it before treating the refusal as a standing constraint. **Second, and the reason this was found at all: 'no reference exists' is a claim about where you looked.** The parity work asserted that no Studio Pro-authored SOAP document existed to pin against and used that to justify mirroring legacy; one existed in a separate repo the whole time (ako/TestApp, which carries both a consumed client and a published service). Byte-parity with what ships is a legitimate goal for a change scoped to stopping a silent drop \u2014 it is NOT evidence the shape is right, and conflating the two is how six defects got a passing test. Where a reference project exists, name it in the code so the next reader does not repeat the search", "refs": []} +{"area": "mdl/backend", "date": "2026-09-11", "symptom": "Making SOAP calls describe structurally instead of as base64 silently rewrote Studio Pro's own documents: a describe -> exec round trip over ako/TestApp flipped `Range.SingleObject` false -> true with no error at all, and turned Clients.SaveOrder's `DataTypes$BooleanType` result into VoidType, which mxbuild reported as `[CE0366]` + `[CE6011]`", "cause": "`webServiceActionRequiresRawBSON` admitted keys BY NAME. That was sound while the supported set was the only nine keys the writer emitted \u2014 but a real call carries FIFTEEN, and six of the new ones are boilerplate mxcli writes at ONE fixed value. Admitting `HttpConfiguration`, `RequestHeaderHandling`, `IsValidationRequired`, `ProxyConfiguration`, `RequestProxyType` and `NewResultHandling` by name meant any call configured beyond mxcli's defaults would be normalised on the next exec", "file": "`mdl/backend/modelsdk/microflow_read_actions.go`, `sdk/mpr/parser_microflow_actions.go` (webServiceActionRequiresRawBSON and its value predicates)", "insight": "**The question a raw-fallback gate answers is not 'do I know this key' but 'would writing this back produce the same document'.** The two coincide only while the writer emits exactly the supported set; the moment a feature lands that widens what is representable, the by-name test starts approving documents it cannot reproduce \u2014 and the loss is invisible, because the result is a VALID model that differs from the user's. **The round trip is the only thing that catches it**: unit tests on the new feature all passed, `mx check` on the newly-written calls was 0 errors, and the regression only appeared when describe -> exec was run over the REFERENCE documents and the BSON diffed (`mxcli bson dump --type microflow --object`, ids normalised away). Two of the five diffs that surfaced were pre-existing microflow describe drift (NoCase case values, bezier control vectors) and unrelated \u2014 worth separating before blaming the change. Fixing it also SHRANK the feature's reach honestly: Studio Pro's own SOAP calls keep the raw form until `Range.SingleObject` is explained, and only mxcli-authored calls describe structurally. **A result type can come from somewhere MDL cannot see** \u2014 SaveOrder's Boolean is the WSDL operation's return type, not an import mapping's entity \u2014 so 'binds a result' does not imply 'derivable'", "refs": []} diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index d6b5b734b..95cac460f 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -587,3 +587,4 @@ {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A SOAP `call web service` naming a mapping and service that REALLY EXIST produced a project Mendix could not LOAD: `mx check` stopped before validation with `Mendix.Modeler.Storage.StorageLoadException \u2026 The text 'c2d1682f-09de-4cc7-95a3-82d5ee5ef243' is not a valid ImportMappingIdentifier`. With the load fixed, the same call was still invalid: CE0386 'Operation GetOrder does not exist in consumed web service Clients.OrderSoapClient' (the operation does exist)", "cause": "Two independent defects, both about a name. (1) `resolveMappingRefForWrite` converted the receive mapping's qualified name to the mapping unit's `$ID`; ImportMappingCall.ReturnValueMapping is an ImportMappingIdentifier \u2014 a qualified name \u2014 so a UUID there is unloadable, not merely invalid. (2) ServiceName was derived as the local part of the imported service's qualified name ('OrderSoapClient'), but it is the WSDL `` ('OrdersWS'); Mendix resolves the operation WITHIN the named service, so a wrong one hides every operation", "file": "`mdl/executor/cmd_microflows_builder_calls.go`, `mdl/executor/webservice_names.go` (new)", "insight": "**A green gate can be green BECAUSE the fixture is broken.** The UUID substitution only happened when the mapping lookup SUCCEEDED. The only SOAP fixture (mdl-examples/doctype-tests/06b-soap-examples.mdl) names a service and mappings that do not exist \u2014 deliberately, to demonstrate dangling references \u2014 so exec took the fallback every time and the qualified name survived. A VALID reference was the single input that triggered an unloadable project, and no test used one. When a code path branches on 'did the lookup resolve', the fixture must cover BOTH branches; a fixture built to show error handling covers only one. **Second: fixing one name unmasks the next error, so work the chain against a real project rather than declaring victory at the first green.** Measured on ako/TestApp (11.14.0, baseline 0 errors), the same one-statement script went StorageLoadException -> CE0386 -> CE0243+CE0366+CE0178 as ReturnValueMapping, then ServiceName were fixed; each error was hidden by the one before it. **Third, two API traps found by debugging rather than reading**: `ListRawUnitsByType` matches the $Type EXACTLY despite its parameter being named typePrefix ('WebServices$ImportedService' returns 0, 'WebServices$ImportedServiceImpl' returns 1) \u2014 which is why the pre-existing resolveWebServiceReference, asking for 'WebServices$ImportedWebService', resolves nothing on any real project; and a unit unmarshalled into map[string]any nests sub-documents as maps, not bson.D, so a lookup asserting bson.D finds nothing, which is indistinguishable from 'no such service' and falls back to the wrong answer instead of failing", "refs": []} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A SOAP `call web service` assigning its result was rejected twice over: `[CE0243] \"The mapping used to return a value of type 'Nothing', but now returns a value of type 'Clients.Order'\"` and `[CE0366] \"Cannot store in variable when there is no return value\"` \u2014 on a call whose receive mapping plainly produces an entity", "cause": "Both engines wrote the result handling's VariableType as `DataTypes$VoidType` unconditionally. Void means the call returns nothing, so it contradicts the mapping AND makes the assignment illegal. Studio Pro writes the entity the mapping produces: `DataTypes$ObjectType{Entity: \"Clients.Order\"}`, which is the Entity of the import mapping's ROOT `ImportMappings$ObjectMappingElement`", "file": "`mdl/executor/webservice_names.go` (resolveImportMappingEntity), `sdk/mpr/writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**Fixing one wrong name in a SOAP call reveals the next; work the chain against a real project instead of stopping at the first green.** On ako/TestApp (11.14.0, baseline 0 errors) the SAME one-statement script went StorageLoadException (ReturnValueMapping written as a UUID) -> CE0386 (ServiceName derived instead of read) -> CE0243+CE0366 (VariableType Void) -> CE0178 (operation arguments), four rounds, each error invisible until the previous fix landed. Any of them could have been called 'the' bug. **The enabler each time was reading the referenced DOCUMENT rather than deriving from the statement**: the WSDL service name is in `Description.Services[].Name` of the imported service, the result entity in `Elements[0].Entity` of the import mapping \u2014 both structured, neither needing the embedded WSDL to be parsed. `ListRawUnitsByType` reaches them on both engines with no new backend method, but note it matches the $Type EXACTLY despite the parameter being named typePrefix, and the types are `WebServices$ImportedServiceImpl` and `ImportMappings$ImportMapping` (NOT the `Mappings$` prefix their child elements use). **Every resolver returns \"\" rather than guessing** \u2014 unresolvable, ambiguous, or wrong-shaped all fall back to what shipped, because a made-up name reproduces the same error with different text in it and is harder to recognise. Remaining and measured: CE0178 needs operation arguments, which MDL cannot express at all (callWebServiceStatement has no argument list), and Range.SingleObject differs from Studio Pro with no error yet attached \u2014 the reference mapping roots carry MaxOccurs 1 while the calls carry SingleObject false, so it is not the mapping's cardinality and would be a guess", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "A DESCRIBE-side resolver for SOAP references had been unreachable since it was written, and its unit test passed. `describe microflow` printed the right service and mapping names throughout", "cause": "`resolveWebServiceReference` looked the service up by ELEMENT ID among units of type `WebServices$ImportedWebService`. It could not match on two independent counts: the stored $Type is `WebServices$ImportedServiceImpl` (ImportedWebService is the SDK name, and nothing is stored under it), and the value compared was never an id \u2014 ImportedService is a BY_NAME_REFERENCE, so it already held `Clients.OrderSoapClient`. Same for the two mapping resolvers. Every call fell through to a fallback that returned the stored string, which is the correct answer", "file": "`mdl/executor/cmd_microflows_format_action.go` (formatWebServiceCallAction; the five resolvers removed)", "insight": "**A resolver whose fallback is the correct answer is indistinguishable in its output from one that works \u2014 so only the input side can prove it runs.** Nothing in the DESCRIBE text could ever have been wrong, which is why this survived: the test that covered it, `TestFormatAction_WebServiceCallResolvesKnownReferences`, built a world where ServiceID WAS a unit id and the unit type WAS `ImportedWebService`, neither of which occurs in any project \u2014 the green test asserted the fiction, not the code. The replacement inverts it: the mock backend calls `t.Fatal` if it is consulted at all, so the test fails unless no lookup happens. **The wider fact, measured, is that the structured DESCRIBE branch is unreachable for real SOAP calls anyway**: all three of ako/TestApp's carry 15 keys and `webServiceActionRequiresRawBSON` supports 9, and mxcli's own writer emits the same 15, so every SOAP call on either engine describes as `call web service raw ''`. A branch nothing reaches cannot be validated by any amount of passing tests over it. **Check a BY_NAME_REFERENCE before writing a resolver**: `modelsdk/gen/*/refs.go` states the kind (`codec.RefByName` here), and an existing test two packages away was already passing `Mod.Service` as the value", "refs": []} +{"area": "mdl/executor", "date": "2026-09-11", "symptom": "`send mapping X` on a SOAP call parsed, `exec` reported success, and the mapping name appeared ZERO times in the written document \u2014 on both engines. mxbuild then rejected the call as `[CE0369] \"Cannot use simple request body, as the operation's body is complex\"`. Separately, an operation taking parameters had no MDL syntax at all and built as `[CE0178] \"Body parameter mapping needs to be refreshed.\"`", "cause": "Both writers emitted an unconditional empty `Microflows$SimpleRequestHandling` for `RequestBodyHandling`. It is a POLYMORPHIC child holding EITHER the operation's arguments (SimpleRequestHandling + WebServiceOperationSimpleParameterMapping entries) or an export mapping (`Microflows$MappingRequestHandling`, NOT the `Mendix$AdvancedRequestHandling` legacy's comment named \u2014 that type is in none of the three ako/TestApp reference documents). The reader compounded it by looking for `RequestHandling`/`ExportMappingCall`, keys no real document carries; the stored key is `RequestBodyHandling`", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go`, `sdk/mpr/writer_microflow_actions.go` (webServiceRequestBody), `mdl/executor/webservice_names.go` (webServiceParameterPath), `mdl/grammar/domains/MDLMicroflow.g4`", "insight": "**Two gaps that look independent can be one polymorphic property, and finding that out is what makes the validation possible.** Arguments and the send mapping are the two branches of `RequestBodyHandling`, so the mutual-exclusion rule (MDL-SOAP01) only becomes enforceable once BOTH exist \u2014 implementing either alone means `check` can refuse a combination it cannot offer an alternative to. **The stored ParameterPath is derivable, which is what makes readable syntax possible**: `escape(operation.RequestBodyElementName) + \"|\" + name` reads the element off `Description.Services[].Operations[]` of the imported service, so MDL says `OrderId` and not `http%3A//www.example.com/:GetOrder|OrderId`. Escaping is per SEGMENT \u2014 `:` inside a segment becomes %3A, the separator `:` and the `/`es are left alone \u2014 and a segment containing `%` is REFUSED because Mendix's escaping of it is unmeasured. **The typed-array marker is the silent trap**: a populated ParameterMappings list leads with marker 2 and `codec.lookupListMarker` DEFAULTS TO 3, so without a `RegisterListMarker` the arguments serialize under the wrong array version \u2014 invisible to mxbuild, fatal to Studio Pro. Control: stubbing the registration makes the test report `marker = 3, want int32(2)`", "refs": []} diff --git a/.claude/skills/mendix/write-microflows/reference/integration.md b/.claude/skills/mendix/write-microflows/reference/integration.md index de1c73da7..f45823dc6 100644 --- a/.claude/skills/mendix/write-microflows/reference/integration.md +++ b/.claude/skills/mendix/write-microflows/reference/integration.md @@ -12,7 +12,7 @@ round-trip without dropping SOAP actions. -- Structured form. Resolved SOAP references use normal qualified names. $Root = call web service SampleSOAP.OrderService operation FetchSampleItems -send mapping SampleSOAP.OrderRequest +send mapping SampleSOAP.OrderRequest from $Request receive mapping SampleSOAP.OrderResponse timeout 30 on error rollback; @@ -20,13 +20,63 @@ on error rollback; -- Quoted raw IDs are accepted when old project references are dangling or unavailable. $Root = call web service 'sample-service-id' operation FetchSampleItems -send mapping 'sample-send-mapping-id' receive mapping 'sample-receive-mapping-id'; -- Raw escape hatch emitted for unsupported SOAP fields. $Root = call web service raw 'AQID'; ``` +### The request body: arguments OR a send mapping, never both + +A call stores **one** request body (`Microflows$RequestBodyHandling`), so the two +forms are alternatives. Asking for both is refused as **MDL-SOAP01** by +`mxcli check` and by `exec` — the same function runs in each. + +**Arguments** bind the operation's parameters, in the same `(Name = value)` form +every other call statement uses: + +```mdl +$Order = call web service Clients.OrderSoapClient +operation GetOrder (OrderId = $Customer/OrderId) +receive mapping Clients.SoapOrdersImportMapping; +``` + +Mendix stores each one under a `ParameterPath` +(`http%3A//www.example.com/:GetOrder|OrderId`) built from the operation's request +body element. mxcli reads that element off the consumed service document and +builds the path, so the script names only the parameter. Two consequences: + +- **The consumed service must be present and declare the operation.** An + operation mxcli cannot resolve is refused rather than written with a made-up + path — a wrong path reproduces the same error with different text in it. +- **A misspelled parameter name cannot be caught by `mxcli check`.** The names + live in the WSDL's inline schema, which mxcli does not parse; the error arrives + from mxbuild as **CE0178** "Body parameter mapping needs to be refreshed" — + which is also what you get if you omit arguments an operation requires. + +**A send mapping** builds the whole body from an export mapping, and needs the +variable it maps **from**: + +```mdl +call web service Clients.OrderSoapClient +operation SaveOrder +send mapping Clients.SoapOrderExportMapping from $NewSaveOrder; +``` + +`from $var` is not optional. An export mapping maps an object and Mendix stores +which one; without it the call builds as **CE0369** "Cannot use simple request +body, as the operation's body is complex". + +### Why DESCRIBE sometimes still shows base64 + +`describe microflow` renders a SOAP call structurally only when re-executing that +MDL would reproduce the stored document exactly. A call configured beyond what +MDL spells — HTTP authentication, a custom location, SOAP headers, a per-parameter +export mapping, or a result typed from the WSDL rather than from an import +mapping — keeps the `call web service raw ''` form, which round-trips +byte for byte. That is deliberate: rendering it structurally would silently +normalise the call on the next `exec`. + **Design note:** the raw payload is base64-encoded BSON for the complete action and is authoritative on re-exec. Treat this as round-trip support, not a recommended authoring format for new integrations. diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 1e4976f2d..622e29efd 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -235,6 +235,61 @@ func init() { SeeAlso: []string{"rest", "rest.consumed", "microflow"}, }) + // ── SOAP (legacy web services) ──────────────────────────────────── + + Register(SyntaxFeature{ + Path: "soap", + Summary: "Legacy SOAP web service calls: operations, arguments, send/receive mappings", + Keywords: []string{ + "soap", "web service", "call web service", "wsdl", + "imported service", "consumed web service", "operation", + "send mapping", "receive mapping", "request body", + "parameter path", "ce0178", "ce0369", "ce0386", + }, + Syntax: "[$Var =] CALL WEB SERVICE Module.ImportedService\n" + + " [OPERATION Name [(Param = expr, ...)]]\n" + + " [SEND MAPPING Module.ExportMapping FROM $var]\n" + + " [RECEIVE MAPPING Module.ImportMapping]\n" + + " [TIMEOUT expr]\n" + + " [ON ERROR ...];\n\n" + + "[$Var =] CALL WEB SERVICE RAW '';\n\n" + + "-- The REQUEST BODY is one of two things, never both: the operation's\n" + + "-- arguments, or an export mapping. Mendix stores ONE\n" + + "-- RequestBodyHandling, so asking for each is refused as MDL-SOAP01 by\n" + + "-- `mxcli check` and by exec, which run the same function.\n" + + "--\n" + + "-- ARGUMENTS use the same (Name = value) form as every other call\n" + + "-- statement. Mendix keys each one by a ParameterPath\n" + + "-- (http%3A//www.example.com/:GetOrder|OrderId) built from the\n" + + "-- operation's request body element; mxcli reads that off the consumed\n" + + "-- service document, so the script names only the parameter. The\n" + + "-- service must therefore be present and declare the operation — one it\n" + + "-- cannot resolve is refused, not written with a guessed path. Omitting\n" + + "-- arguments an operation requires is CE0178.\n" + + "--\n" + + "-- A misspelled PARAMETER NAME cannot be checked: the names live in the\n" + + "-- WSDL's inline schema, which mxcli does not parse. It arrives as\n" + + "-- CE0178 from mxbuild.\n" + + "--\n" + + "-- SEND MAPPING needs FROM $var — an export mapping maps an object and\n" + + "-- Mendix stores which one. Without it the call builds as CE0369.\n" + + "--\n" + + "-- DESCRIBE renders the structured form only when re-executing it would\n" + + "-- reproduce the stored document exactly. A call with HTTP\n" + + "-- authentication, a custom location, SOAP headers, a per-parameter\n" + + "-- export mapping, or a result typed from the WSDL rather than from an\n" + + "-- import mapping keeps the RAW form, which round-trips byte for byte.", + Example: "-- Arguments\n" + + "$Order = call web service Clients.OrderSoapClient\n" + + " operation GetOrder (OrderId = $Customer/OrderId)\n" + + " receive mapping Clients.SoapOrdersImportMapping;\n\n" + + "-- Export mapping as the request body\n" + + "call web service Clients.OrderSoapClient\n" + + " operation SaveOrder\n" + + " send mapping Clients.SoapOrderExportMapping from $NewSaveOrder;", + SeeAlso: []string{"integration", "microflow", "rest"}, + }) + Register(SyntaxFeature{ Path: "rest.consumed", Summary: "Create consumed REST clients with operations, mappings, and authentication", diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 2a625246b..79eb76668 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -520,7 +520,14 @@ it is for pages. | Call JS action | `$Result = call javascript action Module.Name (Param = $value);` | JavaScript action (nanoflow/microflow) | | Call Java action | `$Result = call java action Module.Name (Param = $value);` | Java action (microflow only) | | Call web service | `$Result = call web service Module.Service operation OperationName;` | Legacy SOAP; quoted refs are fallback for dangling raw IDs | +| Call web service (arguments) | `$Result = call web service Module.Service operation GetOrder (OrderId = $Id) receive mapping Module.IMM;` | Binds the operation's parameters, same `(Name = value)` form as every other call. mxcli builds the stored `ParameterPath` from the operation, so the script names only the parameter. Needs the consumed service present — an operation it cannot resolve is refused, not guessed. Without them an operation that takes parameters is **CE0178** | +| Call web service (send mapping) | `call web service Module.Service operation SaveOrder send mapping Module.EMM from $Order;` | Request body built by an export mapping. `from $var` is **required** — Mendix stores which object is mapped, and without it the call is **CE0369** | | Call web service raw | `$Result = call web service raw 'base64-bson';` | Escape hatch for byte-for-byte legacy SOAP round-trip | + +> **A call has ONE request body.** Arguments and a send mapping are alternatives — +> Mendix stores one `RequestBodyHandling` — so a statement asking for both is +> refused as **MDL-SOAP01** by `mxcli check` and by `exec`, which call the same +> function. | REST call (string) | `$Var = rest call get '' returns string;` | Body as string | | REST call (response) | `$Var = rest call get '' returns response;` | `System.HttpResponse` object. There is no specialization form — Mendix does not allow HttpResponse to be specialized (CE1540) | | REST call (file document) | `$Var = rest call get '' returns Module.MyFile;` | Stores the body in a file document. Must be a **specialization** of `System.FileDocument` — the base type is rejected as a return type (CE0362 / MDL064) | diff --git a/mdl-examples/doctype-tests/06b-soap-examples.mdl b/mdl-examples/doctype-tests/06b-soap-examples.mdl index 0b4ef17af..0e32f1833 100644 --- a/mdl-examples/doctype-tests/06b-soap-examples.mdl +++ b/mdl-examples/doctype-tests/06b-soap-examples.mdl @@ -5,12 +5,19 @@ create entity SampleSOAP.OrderResponse ( ); / -create microflow SampleSOAP.ACT_FetchItems () +-- A call's REQUEST BODY is one of two things, never both: the operation's +-- arguments, or an export mapping. Mendix stores one RequestBodyHandling, so +-- writing both is refused as MDL-SOAP01. + +-- Form 1: an export mapping, with the variable it maps FROM. The variable is +-- not optional — without it Mendix has no object to map and the call builds as +-- CE0369 "Cannot use simple request body, as the operation's body is complex". +create microflow SampleSOAP.ACT_FetchItems ($Request : SampleSOAP.OrderResponse) returns SampleSOAP.OrderResponse as $Root begin $Root = call web service SampleSOAP.OrderService operation FetchSampleItems - send mapping SampleSOAP.OrderRequest + send mapping SampleSOAP.OrderRequest from $Request receive mapping SampleSOAP.OrderResponse timeout 30 on error rollback; @@ -19,6 +26,27 @@ begin end; / +-- Form 2: the operation's arguments, in the same (Name = value) form every +-- other call statement uses. Each one binds a named parameter of the operation; +-- mxcli builds the stored ParameterPath from the operation's request body +-- element, so the script says only the parameter name. +-- +-- This form needs the consumed service to be present and to declare the +-- operation — there is no path to build otherwise, and exec refuses rather than +-- inventing one. That is why this example is the dangling-reference one's +-- opposite and cannot be written against SampleSOAP.OrderService here; see +-- mdl-examples/bug-tests for a run against a real WSDL. +create microflow SampleSOAP.ACT_FetchItemsNoBody () +returns SampleSOAP.OrderResponse as $Root +begin + $Root = call web service SampleSOAP.OrderService + operation FetchSampleItems + receive mapping SampleSOAP.OrderResponse; + + return $Root; +end; +/ + create microflow SampleSOAP.ACT_FetchItemsDanglingRefs () returns SampleSOAP.OrderResponse as $Root begin diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index ed073e20c..ef7b7ed51 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -650,15 +650,20 @@ func (s *CallJavaScriptActionStmt) isMicroflowStatement() {} // CallWebServiceStmt represents a legacy SOAP web service call. type CallWebServiceStmt struct { - OutputVariable string // Optional output variable - RawBSONBase64 string // Raw Microflows$CallWebServiceAction BSON for lossless roundtrip - ServiceID string // Consumed web service ID or qualified name - OperationName string // Operation name - SendMappingID string // Optional export mapping ID or qualified name - ReceiveMappingID string // Optional import mapping ID or qualified name - Timeout Expression // Optional timeout expression - ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + OutputVariable string // Optional output variable + RawBSONBase64 string // Raw Microflows$CallWebServiceAction BSON for lossless roundtrip + ServiceID string // Consumed web service ID or qualified name + OperationName string // Operation name + Arguments []CallArgument // Optional operation arguments — Microflows$SimpleRequestHandling + SendMappingID string // Optional export mapping ID or qualified name + // SendMappingVariable is the variable the export mapping maps FROM. An + // export mapping always maps an object, so a send mapping without one is + // incomplete — Mendix stores it as MappingRequestHandling.MappingVariableName. + SendMappingVariable string + ReceiveMappingID string // Optional import mapping ID or qualified name + Timeout Expression // Optional timeout expression + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation } func (s *CallWebServiceStmt) isMicroflowStatement() {} diff --git a/mdl/backend/modelsdk/microflow_read_actions.go b/mdl/backend/modelsdk/microflow_read_actions.go index 2be303b56..2b4a02b82 100644 --- a/mdl/backend/modelsdk/microflow_read_actions.go +++ b/mdl/backend/modelsdk/microflow_read_actions.go @@ -3,6 +3,8 @@ package modelsdkbackend import ( + "strings" + "go.mongodb.org/mongo-driver/v2/bson" "github.com/mendixlabs/mxcli/model" @@ -374,11 +376,17 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { out.ReceiveMappingID = model.ID(rawStr(imc, "ReturnValueMapping")) } } + // RequestHandling / ExportMappingCall is a shape NO reference document + // carries — all three of ako/TestApp's calls store RequestBodyHandling — + // so SendMappingID was never populated from a real project. Kept because + // removing it would be a guess in the other direction, and read the real + // key below. if rqh, ok := raw.Lookup("RequestHandling").DocumentOK(); ok { if emc, ok := rqh.Lookup("ExportMappingCall").DocumentOK(); ok { out.SendMappingID = model.ID(rawStr(emc, "Mapping")) } } + readWebServiceRequestBody(raw, out) if webServiceActionRequiresRawBSON(raw) { out.RawBSON = raw } @@ -898,30 +906,212 @@ func readMappingCall(doc, imc bson.Raw) (h *microflows.ResultHandlingMapping, fo } // webServiceActionRequiresRawBSON reports whether a CALL WEB SERVICE action -// carries any field the structured describe form can't represent, in which case +// carries anything the structured describe form can't reproduce, in which case // the renderer emits the `call web service raw ''` fallback. Mirrors the -// legacy webServiceActionRequiresRawBSON supported-key set exactly so both engines -// agree on when to fall back. +// legacy webServiceActionRequiresRawBSON decision exactly so both engines agree +// on when to fall back. +// +// The question is NOT "is this key known" but "would writing this back from the +// structured form produce the same document". Until the request body became +// authorable the two were the same, because the nine keys below were the only +// ones the writer emitted. They are not the same now: a real call carries +// fifteen, and six of the new ones are boilerplate mxcli writes at ONE fixed +// value. Admitting them by name would silently normalise a call with HTTP +// authentication, a custom location or a non-default proxy the moment anyone +// ran describe → exec on it — which is the silent drop this whole change exists +// to remove, reintroduced one layer up. +// +// So each of those six is admitted only AT that value. Anything else keeps the +// raw fallback, which round-trips byte for byte. func webServiceActionRequiresRawBSON(raw bson.Raw) bool { - supported := map[string]bool{ + // Keys the structured form carries in full, whatever their value. + represented := map[string]bool{ "$ID": true, "$Type": true, "ErrorHandlingType": true, "ImportedService": true, "OperationName": true, "TimeOutExpression": true, - "UseRequestTimeOut": true, - "NewResultHandling": true, "RequestHandling": true, + // ServiceName is not written by DESCRIBE and does not need to be: the + // write path re-reads it from the imported service document, which is + // the authoritative source (that is the CE0386 fix). A call whose + // service cannot be resolved is already broken. + "ServiceName": true, } els, err := raw.Elements() if err != nil { return true } for _, el := range els { - if !supported[el.Key()] { - return true + key := el.Key() + if represented[key] { + continue + } + if ok, known := webServiceFixedValueIsDefault(raw, key); known { + if !ok { + return true + } + continue } + return true + } + return false +} + +// webServiceFixedValueIsDefault reports whether one of the keys mxcli writes at a +// FIXED value currently holds that value. known is false for a key it does not +// judge, which the caller treats as unrepresentable. +func webServiceFixedValueIsDefault(raw bson.Raw, key string) (ok, known bool) { + switch key { + case "IsValidationRequired": + v, isBool := raw.Lookup(key).BooleanOK() + return isBool && !v, true + case "UseRequestTimeOut": + v, isBool := raw.Lookup(key).BooleanOK() + return isBool && v, true + case "RequestProxyType": + return rawStr(raw, key) == "DefaultProxy", true + case "ProxyConfiguration": + return raw.Lookup(key).Type == bson.TypeNull, true + case "HttpConfiguration": + doc, isDoc := raw.Lookup(key).DocumentOK() + return isDoc && isDefaultWebServiceHTTPConfig(doc), true + case "RequestHeaderHandling": + doc, isDoc := raw.Lookup(key).DocumentOK() + return isDoc && isEmptySimpleRequestHandling(doc), true + case "RequestBodyHandling": + doc, isDoc := raw.Lookup(key).DocumentOK() + return isDoc && webServiceRequestBodyIsRepresentable(doc), true + case "NewResultHandling": + doc, isDoc := raw.Lookup(key).DocumentOK() + return isDoc && webServiceResultHandlingIsRepresentable(doc), true + } + return false, false +} + +// webServiceResultHandlingIsRepresentable reports whether a call's result +// handling is one the writer reproduces exactly. +// +// This key used to be admitted by name, which was safe only while the six +// boilerplate keys kept every real call on the raw path anyway. A describe → +// exec round trip over ako/TestApp caught both ways it is not: +// +// - Clients.SaveOrder binds $IsSaved with NO import mapping, and its +// VariableType is DataTypes$BooleanType — the OPERATION's own return type, +// which lives in the WSDL and is therefore not derivable from MDL. Written +// back as VoidType it is two errors: CE0366 "Cannot store in variable when +// there is no return value" and CE6011 "The type of the output variable does +// not match the return type of the operation." +// - Clients.GetOrders carries Range.SingleObject false where mxcli writes true +// — the one divergence from the reference documents still unexplained. No +// error comes of it, which is precisely why it must not be written silently: +// the round trip would change the user's document with nothing to show for it. +// +// So a result handling is representable only when the writer's fixed values are +// already the stored ones. mxcli's own calls qualify; Studio Pro's do not, and +// keep the byte-exact raw fallback until SingleObject is settled. +func webServiceResultHandlingIsRepresentable(doc bson.Raw) bool { + if rawStr(doc, "$Type") != "Microflows$ResultHandling" { + return false + } + bound := rawStr(doc, "ResultVariableName") != "" + if bind, ok := doc.Lookup("Bind").BooleanOK(); !ok || bind != bound { + return false + } + imc, hasMapping := doc.Lookup("ImportMappingCall").DocumentOK() + vt, hasType := doc.Lookup("VariableType").DocumentOK() + if !hasType { + return false + } + if !hasMapping { + // No mapping: the writer emits VoidType, so only VoidType round-trips. + return rawStr(vt, "$Type") == "DataTypes$VoidType" + } + if rawStr(vt, "$Type") != "DataTypes$ObjectType" { + return false + } + if rawStr(imc, "$Type") != "Microflows$ImportMappingCall" || + rawStr(imc, "Commit") != "YesWithoutEvents" || + rawStr(imc, "ContentType") != "Xml" || + rawStr(imc, "ObjectHandlingBackup") != "Create" || + rawStr(imc, "ParameterVariableName") != "" || + rawStr(imc, "ReturnValueMapping") == "" { + return false + } + if force, ok := imc.Lookup("ForceSingleOccurrence").BooleanOK(); !ok || force { + return false + } + rng, ok := imc.Lookup("Range").DocumentOK() + if !ok || rawStr(rng, "$Type") != "Microflows$ConstantRange" { + return false + } + single, ok := rng.Lookup("SingleObject").BooleanOK() + return ok && single +} + +// isDefaultWebServiceHTTPConfig reports whether an HttpConfiguration is the one a +// SOAP call gets when nothing is configured — the only one mxcli writes. +func isDefaultWebServiceHTTPConfig(doc bson.Raw) bool { + if rawStr(doc, "$Type") != "Microflows$HttpConfiguration" { + return false + } + for _, key := range []string{"ClientCertificate", "CustomLocation", + "HttpAuthenticationPassword", "HttpAuthenticationUserName"} { + if rawStr(doc, key) != "" { + return false + } + } + if doc.Lookup("CustomLocationTemplate").Type != bson.TypeNull { + return false + } + if rawStr(doc, "HttpMethod") != "Post" { + return false + } + for key, want := range map[string]bool{"OverrideLocation": false, "UseHttpAuthentication": false} { + v, isBool := doc.Lookup(key).BooleanOK() + if !isBool || v != want { + return false + } + } + return len(rawDocElements(doc, "HttpHeaderEntries")) == 0 +} + +// isEmptySimpleRequestHandling reports whether a request handling is the bare +// Simple form — no parameter mappings — which is all mxcli writes for headers. +func isEmptySimpleRequestHandling(doc bson.Raw) bool { + return rawStr(doc, "$Type") == "Microflows$SimpleRequestHandling" && + rawStr(doc, "NullValueOption") == "LeaveOutElement" && + len(rawDocElements(doc, "ParameterMappings")) == 0 +} + +// webServiceRequestBodyIsRepresentable reports whether a RequestBodyHandling is +// one MDL can spell: an export mapping, or simple parameter mappings whose names +// survive the round trip. +func webServiceRequestBodyIsRepresentable(doc bson.Raw) bool { + switch rawStr(doc, "$Type") { + case "Microflows$MappingRequestHandling": + // All three properties are carried on the model and restated by MDL. + return rawStr(doc, "MappingId") != "" && rawStr(doc, "MappingVariableName") != "" + case "Microflows$SimpleRequestHandling": + if rawStr(doc, "NullValueOption") != "LeaveOutElement" { + return false + } + for _, pm := range rawDocElements(doc, "ParameterMappings") { + // An ADVANCED mapping (a per-parameter export mapping) has no MDL + // spelling at all, and a non-empty ParameterName is a shape no + // reference document carries, so neither is judged representable. + if rawStr(pm, "$Type") != "Microflows$WebServiceOperationSimpleParameterMapping" || + rawStr(pm, "ParameterName") != "" { + return false + } + // The parameter name MDL spells is the path's last segment; without + // one the write path could not rebuild the same path. + if !strings.Contains(rawStr(pm, "ParameterPath"), "|") { + return false + } + } + return true } return false } @@ -1156,6 +1346,60 @@ func errorHandlingTypeOf(el element.Element) string { // Element 0 of a Mendix array is a version marker (an int32), never data, so it // simply fails the document check and is skipped — the same shape the mapping // readers above rely on. +// readWebServiceRequestBody reads a SOAP call's RequestBodyHandling — the +// polymorphic child holding EITHER the operation's arguments or an export +// mapping — back into the semantic model. +// +// Dispatched on $Type, never on which fields happen to be present: the two +// variants differ in arity, and assigning whichever keys are there would +// quietly turn one into the other. +// +// The parameter NAME is recovered from the stored ParameterPath's last segment, +// which is what MDL spells. A path with no "|" yields no name, so the argument +// is read with an empty Name and the describe path renders the action raw — +// better than inventing a name that would write a different path back. +func readWebServiceRequestBody(raw bson.Raw, out *microflows.WebServiceCallAction) { + body, ok := raw.Lookup("RequestBodyHandling").DocumentOK() + if !ok { + return + } + switch rawStr(body, "$Type") { + case "Microflows$MappingRequestHandling": + // STORAGE NAMES: MappingId / MappingVariableName. gen binds the same two + // as Mapping / MappingArgumentVariableName — both listed in its key + // audit — so a reader keyed on gen's names finds nothing here. + out.SendMappingID = model.ID(rawStr(body, "MappingId")) + out.SendMappingVariable = rawStr(body, "MappingVariableName") + out.SendMappingContentType = rawStr(body, "ContentType") + case "Microflows$SimpleRequestHandling": + for _, pm := range rawDocElements(body, "ParameterMappings") { + if rawStr(pm, "$Type") != "Microflows$WebServiceOperationSimpleParameterMapping" { + // An advanced (per-parameter export mapping) entry, which MDL + // cannot author. Read nothing rather than half of it; the raw + // fallback carries the action. + continue + } + path := rawStr(pm, "ParameterPath") + name := "" + if i := strings.LastIndex(path, "|"); i >= 0 { + name = path[i+1:] + } + // Absent reads as true: both reference mappings carry true, and a + // bound parameter is by definition one Studio Pro has ticked. + checked, ok := pm.Lookup("IsChecked").BooleanOK() + if !ok { + checked = true + } + out.Arguments = append(out.Arguments, microflows.WebServiceArgument{ + Name: name, + Path: path, + Expression: rawStr(pm, "Argument"), + Checked: checked, + }) + } + } +} + func rawDocElements(raw bson.Raw, key string) []bson.Raw { arr, ok := raw.Lookup(key).ArrayOK() if !ok { diff --git a/mdl/backend/modelsdk/microflow_webservice_write.go b/mdl/backend/modelsdk/microflow_webservice_write.go index 26d597ee2..583f810e6 100644 --- a/mdl/backend/modelsdk/microflow_webservice_write.go +++ b/mdl/backend/modelsdk/microflow_webservice_write.go @@ -60,21 +60,29 @@ import ( // made assigning the result its own error (CE0366). VoidType stays the // fallback for a mapping that cannot be resolved. // -// Two remain: +// Two more have since been fixed, and they turned out to be one thing: +// RequestBodyHandling is a polymorphic child holding EITHER the operation's +// arguments or an export mapping, so what looked like two gaps was the two +// branches of one property. +// +// - Operation ARGUMENTS are Microflows$WebServiceOperationSimpleParameterMapping +// entries inside a SimpleRequestHandling, keyed by an escaped ParameterPath +// ("http%3A//www.example.com/:GetOrder|OrderId" — the operation's +// RequestBodyElementName, escaped, plus "|" plus the parameter). Writing that +// list empty gave CE0178 "Body parameter mapping needs to be refreshed". +// - A SEND MAPPING is a Microflows$MappingRequestHandling, NOT the +// Mendix$AdvancedRequestHandling legacy's comment named (that type appears in +// none of the three reference documents). Writing SimpleRequestHandling +// regardless — which both engines did — dropped the mapping silently and gave +// CE0369 "Cannot use simple request body, as the operation's body is complex". +// +// One remains: // // - Range.SingleObject follows the operation's cardinality; both reference // calls write false where both engines write true. No error has been // measured from it, so it is left until one is — the reference roots carry // MaxOccurs 1 while the calls carry SingleObject false, so it is NOT simply // the mapping's cardinality and would be a guess today. -// - Operation ARGUMENTS are Microflows$WebServiceOperationSimpleParameterMapping -// entries inside RequestBodyHandling.ParameterMappings, keyed by an escaped -// ParameterPath ("http%3A//www.example.com/:GetOrder|OrderId" — the -// operation's RequestBodyElementName, escaped, plus "|" plus the parameter). -// Writing that list empty gives CE0178 "Body parameter mapping needs to be -// refreshed" — now the ONLY error left on a real call. It needs MDL SYNTAX -// before it can be written at all: callWebServiceStatement has no argument -// list, so there is nothing to serialize yet. // // Two shapes are deliberately NOT re-derived here: // @@ -87,15 +95,29 @@ import ( // already carries one such collision: Microflows$HttpHeaderEntry is // registered as 2 in microflow_write.go and as 3 in odata_write.go, and // which one wins is decided by file order.) -// - RequestBodyHandling is always SimpleRequestHandling, even when the -// statement carries a SEND MAPPING — matching legacy, and WRONG: the real -// type is Microflows$MappingRequestHandling (see above). Until that is -// implemented the send mapping is silently dropped on both engines, and -// `call web service raw` is the only way to author one. +// - MappingRequestHandling's two keys are written as RAW strings rather than +// through the gen accessors. gen binds them as `Mapping` and +// `MappingArgumentVariableName`; Studio Pro stores `MappingId` and +// `MappingVariableName` (modelsdk/gen/keyaudit_test.go). A document written +// through gen's names is one mxbuild tolerates and Studio Pro cannot open. // // Every null the document carries is written IN KEY POSITION rather than through // NullFields, for the same reason and with the same consequence — see addNull. +func init() { + // A populated ParameterMappings list leads with marker 2 (measured on all + // three ako/TestApp calls). The codec's default is 3, so WITHOUT this the + // arguments would serialize under the wrong array version — the class of + // defect that makes a project Studio Pro cannot open, and one mxbuild does + // not catch. + // + // Unlike the markers the note above keeps out of the registry, this child + // type is SOAP-only: nothing else writes a + // WebServiceOperationSimpleParameterMapping, so there is no writer for a + // global registration to disturb. + codec.RegisterListMarker("Microflows$WebServiceOperationSimpleParameterMapping", 2) +} + // webServiceCallActionToGen builds a Microflows$CallWebServiceAction. Mirrors // sdk/mpr.serializeWebServiceCallAction field-for-field, in the same key order. func webServiceCallActionToGen(a *microflows.WebServiceCallAction) element.Element { @@ -122,8 +144,10 @@ func webServiceCallActionToGen(a *microflows.WebServiceCallAction) element.Eleme addPart(g, "NewResultHandling", webServiceResultHandlingToGen(a)) addStr(g, "OperationName", a.OperationName) addNull(g, "ProxyConfiguration") - addPart(g, "RequestBodyHandling", simpleRequestHandlingToGen()) - addPart(g, "RequestHeaderHandling", simpleRequestHandlingToGen()) + addPart(g, "RequestBodyHandling", webServiceRequestBodyToGen(a)) + // The HEADER handling is always Simple and always empty: MDL cannot author + // SOAP headers, and all three reference calls carry the bare form. + addPart(g, "RequestHeaderHandling", simpleRequestHandlingToGen(nil)) addStr(g, "RequestProxyType", "DefaultProxy") addStr(g, "ServiceName", webServiceName(a)) addStr(g, "TimeOutExpression", orDefault(a.TimeoutExpression, "300")) @@ -199,12 +223,64 @@ func webServiceVariableType(a *microflows.WebServiceCallAction) element.Element return vt } -// simpleRequestHandlingToGen builds the Microflows$SimpleRequestHandling used for -// both the body and the header handling. -func simpleRequestHandlingToGen() element.Element { +// webServiceRequestBodyToGen builds RequestBodyHandling — the polymorphic child +// that carries EITHER the operation's arguments or an export mapping. +// +// The executor refuses a statement asking for both (MDL-SOAP01), so the branch +// here is a plain else: a send mapping wins only because it cannot coexist with +// arguments, not by precedence. +func webServiceRequestBodyToGen(a *microflows.WebServiceCallAction) element.Element { + if a.SendMappingID != "" { + return mappingRequestHandlingToGen(a) + } + return simpleRequestHandlingToGen(a.Arguments) +} + +// mappingRequestHandlingToGen builds Microflows$MappingRequestHandling — a SOAP +// request body produced by an export mapping. +// +// Both name keys are written RAW. gen binds them as `Mapping` and +// `MappingArgumentVariableName`, which are the SDK names; Studio Pro stores +// `MappingId` and `MappingVariableName`, and both mismatches are listed in +// modelsdk/gen/keyaudit_test.go. mxbuild tolerates the wrong spellings, so the +// symptom of getting this wrong is not a build error — it is a document Studio +// Pro cannot open. +func mappingRequestHandlingToGen(a *microflows.WebServiceCallAction) element.Element { + rh := newElem("Microflows$MappingRequestHandling", "") + // "Json" is what Studio Pro wrote on the one reference document + // (ako/TestApp Clients.SaveOrder) — surprising on an XML protocol, which is + // why a stored value is carried through rather than normalised, and why the + // default is the observed one rather than the plausible one. + addStr(rh, "ContentType", orDefault(a.SendMappingContentType, "Json")) + addStr(rh, "MappingId", string(a.SendMappingID)) + addStr(rh, "MappingVariableName", a.SendMappingVariable) + return rh +} + +// simpleRequestHandlingToGen builds the Microflows$SimpleRequestHandling that +// carries the operation's arguments. Passing nil gives the empty form, which is +// what the header handling and an argument-less call use. +func simpleRequestHandlingToGen(args []microflows.WebServiceArgument) element.Element { rh := newElem("Microflows$SimpleRequestHandling", "") addStr(rh, "NullValueOption", "LeaveOutElement") - addEmptyTypedList(rh, "ParameterMappings", 2) + if len(args) == 0 { + addEmptyTypedList(rh, "ParameterMappings", 2) + return rh + } + children := make([]element.Element, 0, len(args)) + for _, arg := range args { + pm := newElem("Microflows$WebServiceOperationSimpleParameterMapping", "") + addStr(pm, "Argument", arg.Expression) + addBool(pm, "IsChecked", arg.Checked) + // ParameterName is "" in both reference mappings. What fills it is + // unmeasured — plausibly an RPC-style binding, which neither reference + // operation uses — so it is written empty rather than guessed at from + // the argument's own name. + addStr(pm, "ParameterName", "") + addStr(pm, "ParameterPath", arg.Path) + children = append(children, pm) + } + addPartList(rh, "ParameterMappings", children) return rh } diff --git a/mdl/backend/modelsdk/microflow_webservice_write_test.go b/mdl/backend/modelsdk/microflow_webservice_write_test.go index 2a0aa2239..a80139fc3 100644 --- a/mdl/backend/modelsdk/microflow_webservice_write_test.go +++ b/mdl/backend/modelsdk/microflow_webservice_write_test.go @@ -6,6 +6,7 @@ import ( "testing" bsonv1 "go.mongodb.org/mongo-driver/bson" + bsonv2 "go.mongodb.org/mongo-driver/v2/bson" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/modelsdk/codec" @@ -314,3 +315,260 @@ func assertTypedArrayMarker(t *testing.T, doc bsonv1.D, key string, want int32) t.Errorf("%s marker = %#v, want int32(%d)", key, arr[0], want) } } + +// TestWebServiceCallAction_ArgumentsAreSimpleParameterMappings is the regression +// test for CE0178. +// +// An operation taking parameters needs them bound, and an empty +// SimpleRequestHandling.ParameterMappings is mxbuild's "Body parameter mapping +// needs to be refreshed." Measured on 11.14.0 against ako/TestApp: the same +// script goes 1 error -> 0 once the list is written. +// +// The ParameterPath is asserted character for character against what Studio Pro +// stored, because a plausible wrong escaping is exactly what mxbuild accepts and +// Studio Pro does not. +func TestWebServiceCallAction_ArgumentsAreSimpleParameterMappings(t *testing.T) { + a := fullWebServiceCall() + a.Arguments = []microflows.WebServiceArgument{{ + Name: "OrderId", + Path: "http%3A//www.example.com/:GetOrder|OrderId", + Expression: "$Customer/OrderId", + Checked: true, + }} + + body, ok := docGet(encodeMicroflowAction(t, a), "RequestBodyHandling").(bsonv1.D) + if !ok { + t.Fatal("RequestBodyHandling is not a document") + } + if got := docGet(body, "$Type"); got != "Microflows$SimpleRequestHandling" { + t.Fatalf("$Type = %#v, want Microflows$SimpleRequestHandling", got) + } + + arr, ok := docGet(body, "ParameterMappings").(bsonv1.A) + if !ok || len(arr) != 2 { + t.Fatalf("ParameterMappings = %#v, want the marker plus one mapping", docGet(body, "ParameterMappings")) + } + // Marker 2 — measured on all three reference calls. The codec's DEFAULT is + // 3, so without the RegisterListMarker in the writer this is the assertion + // that fails, and a wrong marker is the class of defect that makes a project + // Studio Pro cannot open while mxbuild stays silent. + if got, isInt := arr[0].(int32); !isInt || got != 2 { + t.Errorf("ParameterMappings marker = %#v, want int32(2)", arr[0]) + } + + pm, ok := arr[1].(bsonv1.D) + if !ok { + t.Fatalf("mapping = %#v, want a document", arr[1]) + } + for _, want := range []struct { + key string + val any + }{ + {"$Type", "Microflows$WebServiceOperationSimpleParameterMapping"}, + {"Argument", "$Customer/OrderId"}, + {"IsChecked", true}, + // "" in both reference mappings; what fills it is unmeasured. + {"ParameterName", ""}, + {"ParameterPath", "http%3A//www.example.com/:GetOrder|OrderId"}, + } { + if got := docGet(pm, want.key); got != want.val { + t.Errorf("%s = %#v, want %#v", want.key, got, want.val) + } + } + + // The HEADER handling stays the bare empty form — it is not where arguments go. + hdr, ok := docGet(encodeMicroflowAction(t, a), "RequestHeaderHandling").(bsonv1.D) + if !ok { + t.Fatal("RequestHeaderHandling is not a document") + } + assertTypedArrayMarker(t, hdr, "ParameterMappings", 2) +} + +// TestWebServiceCallAction_SendMappingIsAMappingRequestHandling is the +// regression test for CE0369. +// +// `send mapping` parsed, was accepted and was DISCARDED: the writer emitted an +// empty SimpleRequestHandling regardless, and mxbuild reported "Cannot use +// simple request body, as the operation's body is complex". The mapping name +// appeared zero times in the written document, on either engine. +// +// The two name keys are the ones modelsdk/gen gets wrong (its key audit lists +// Mapping -> MappingId and MappingArgumentVariableName -> MappingVariableName), +// so this test is what stops a future rewrite through the gen accessors: the +// wrong spellings build clean and give a document Studio Pro cannot open. +func TestWebServiceCallAction_SendMappingIsAMappingRequestHandling(t *testing.T) { + a := fullWebServiceCall() + a.ReceiveMappingID = "" + a.OutputVariable = "" + a.UseReturnVariable = false + a.SendMappingID = "Clients.SoapOrderExportMapping" + a.SendMappingVariable = "NewSaveOrder" + + body, ok := docGet(encodeMicroflowAction(t, a), "RequestBodyHandling").(bsonv1.D) + if !ok { + t.Fatal("RequestBodyHandling is not a document") + } + for _, want := range []struct { + key string + val any + }{ + {"$Type", "Microflows$MappingRequestHandling"}, + // "Json" on an XML protocol is what Studio Pro wrote on the one + // reference document — written as observed, not as it would seem. + {"ContentType", "Json"}, + {"MappingId", "Clients.SoapOrderExportMapping"}, + {"MappingVariableName", "NewSaveOrder"}, + } { + if got := docGet(body, want.key); got != want.val { + t.Errorf("%s = %#v, want %#v", want.key, got, want.val) + } + } + // A marker variant carries no Value/ParameterMappings of the other form. + if got := docGet(body, "ParameterMappings"); got != nil { + t.Errorf("ParameterMappings = %#v on a mapping body, want absent", got) + } +} + +// TestWebServiceCallAction_SendMappingContentTypeIsCarried — a stored +// ContentType survives a rewrite rather than being normalised to the default. +// One reference document is not enough to call "Json" the rule. +func TestWebServiceCallAction_SendMappingContentTypeIsCarried(t *testing.T) { + a := fullWebServiceCall() + a.SendMappingID = "M.Export" + a.SendMappingVariable = "Order" + a.SendMappingContentType = "Xml" + + body, _ := docGet(encodeMicroflowAction(t, a), "RequestBodyHandling").(bsonv1.D) + if got := docGet(body, "ContentType"); got != "Xml" { + t.Errorf("ContentType = %#v, want the stored Xml", got) + } +} + +// referenceSoapActionMap is the fifteen-key shape every ako/TestApp SOAP call +// carries, with mxcli's own values for the six boilerplate keys. +// +// It is built as a map and mutated BEFORE marshalling on purpose. Round-tripping +// through bson.Unmarshal to poke at a nested field does not work here: driver v2 +// decodes nested documents into bson.D even when the top level is a bson.M, the +// mirror image of the map-vs-D trap already recorded for driver v1. +func referenceSoapActionMap() bsonv2.M { + return bsonv2.M{ + "$Type": "Microflows$CallWebServiceAction", + "ErrorHandlingType": "Rollback", + "HttpConfiguration": bsonv2.M{ + "$Type": "Microflows$HttpConfiguration", + "ClientCertificate": "", + "CustomLocation": "", + "CustomLocationTemplate": nil, + "HttpAuthenticationPassword": "", + "HttpAuthenticationUserName": "", + "HttpHeaderEntries": bsonv2.A{int32(3)}, + "HttpMethod": "Post", + "OverrideLocation": false, + "UseHttpAuthentication": false, + }, + "ImportedService": "Clients.OrderSoapClient", + "IsValidationRequired": false, + "NewResultHandling": bsonv2.M{ + "$Type": "Microflows$ResultHandling", "Bind": true, + "ImportMappingCall": bsonv2.M{ + "$Type": "Microflows$ImportMappingCall", "Commit": "YesWithoutEvents", + "ContentType": "Xml", "ForceSingleOccurrence": false, + "ObjectHandlingBackup": "Create", "ParameterVariableName": "", + "Range": bsonv2.M{"$Type": "Microflows$ConstantRange", "SingleObject": true}, + "ReturnValueMapping": "Clients.SoapOrdersImportMapping", + }, + "ResultVariableName": "Orders", + "VariableType": bsonv2.M{"$Type": "DataTypes$ObjectType", "Entity": "Clients.Order"}, + }, + "OperationName": "GetOrder", "ProxyConfiguration": nil, + "RequestBodyHandling": bsonv2.M{ + "$Type": "Microflows$SimpleRequestHandling", "NullValueOption": "LeaveOutElement", + "ParameterMappings": bsonv2.A{int32(2), bsonv2.M{ + "$Type": "Microflows$WebServiceOperationSimpleParameterMapping", + "Argument": "2", "IsChecked": true, "ParameterName": "", + "ParameterPath": "http%3A//www.example.com/:GetOrder|OrderId", + }}, + }, + "RequestHeaderHandling": bsonv2.M{ + "$Type": "Microflows$SimpleRequestHandling", "NullValueOption": "LeaveOutElement", + "ParameterMappings": bsonv2.A{int32(2)}, + }, + "RequestProxyType": "DefaultProxy", "ServiceName": "OrdersWS", + "TimeOutExpression": "300", "UseRequestTimeOut": true, + } +} + +func marshalAction(t *testing.T, m bsonv2.M) bsonv2.Raw { + t.Helper() + out, err := bsonv2.Marshal(m) + if err != nil { + t.Fatalf("marshal: %v", err) + } + return out +} + +// TestWebServiceActionRequiresRawBSON_AgreesWithLegacy mirrors the sdk/mpr twin +// case for case. The two engines implement this decision SEPARATELY — one over +// bson.Raw, one over map[string]any — so nothing but a pair of tests keeps them +// from drifting, and a drift here means the same project describes differently +// depending on which engine read it. +func TestWebServiceActionRequiresRawBSON_AgreesWithLegacy(t *testing.T) { + if webServiceActionRequiresRawBSON(marshalAction(t, referenceSoapActionMap())) { + t.Error("an action mxcli would write itself still falls back to raw") + } + + for _, tc := range []struct { + name string + mutit func(bsonv2.M) + }{ + // Measured: Clients.GetOrders stores SingleObject FALSE where mxcli + // writes true. No error comes of it, which is exactly why writing it + // back must not happen silently. + {"Range.SingleObject differs", func(m bsonv2.M) { + m["NewResultHandling"].(bsonv2.M)["ImportMappingCall"].(bsonv2.M)["Range"] = + bsonv2.M{"$Type": "Microflows$ConstantRange", "SingleObject": false} + }}, + // Measured: Clients.SaveOrder binds $IsSaved with NO import mapping and + // a DataTypes$BooleanType — the OPERATION's return type, which lives in + // the WSDL. Written back as VoidType it is CE0366 + CE6011. + {"result type comes from the WSDL, not a mapping", func(m bsonv2.M) { + m["NewResultHandling"] = bsonv2.M{ + "$Type": "Microflows$ResultHandling", "Bind": true, + "ImportMappingCall": nil, + "ResultVariableName": "IsSaved", + "VariableType": bsonv2.M{"$Type": "DataTypes$BooleanType"}, + } + }}, + {"HTTP authentication configured", func(m bsonv2.M) { + m["HttpConfiguration"].(bsonv2.M)["UseHttpAuthentication"] = true + }}, + {"custom location", func(m bsonv2.M) { + m["HttpConfiguration"].(bsonv2.M)["CustomLocation"] = "https://elsewhere/" + }}, + {"a SOAP header is configured", func(m bsonv2.M) { + m["RequestHeaderHandling"].(bsonv2.M)["ParameterMappings"] = bsonv2.A{int32(2), + bsonv2.M{"$Type": "Microflows$WebServiceOperationSimpleParameterMapping"}} + }}, + {"validation required", func(m bsonv2.M) { m["IsValidationRequired"] = true }}, + {"non-default proxy", func(m bsonv2.M) { m["RequestProxyType"] = "NoProxy" }}, + {"timeout disabled", func(m bsonv2.M) { m["UseRequestTimeOut"] = false }}, + {"advanced parameter mapping", func(m bsonv2.M) { + m["RequestBodyHandling"].(bsonv2.M)["ParameterMappings"] = bsonv2.A{int32(2), + bsonv2.M{"$Type": "Microflows$WebServiceOperationAdvancedParameterMapping"}} + }}, + {"parameter path with no name segment", func(m bsonv2.M) { + pms := m["RequestBodyHandling"].(bsonv2.M)["ParameterMappings"].(bsonv2.A) + pms[1].(bsonv2.M)["ParameterPath"] = "http%3A//www.example.com/:GetOrder" + }}, + {"unknown key entirely", func(m bsonv2.M) { m["SomethingNew"] = int32(1) }}, + } { + t.Run(tc.name, func(t *testing.T) { + m := referenceSoapActionMap() + tc.mutit(m) + if !webServiceActionRequiresRawBSON(marshalAction(t, m)) { + t.Error("describes structurally, so a round trip would silently rewrite it") + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index 35b7ab302..d5a437188 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -544,6 +544,11 @@ func isEmptyJavaActionArgument(expr ast.Expression) bool { // addCallWebServiceAction creates a legacy SOAP WebServiceCallAction. func (fb *flowBuilder) addCallWebServiceAction(s *ast.CallWebServiceStmt) model.ID { activityX := fb.posX + // The same function `mxcli check` runs, so the two cannot drift on what a + // valid request body is (MDL-SOAP01). + if err := checkWebServiceRequestBodyStmt(s); err != nil { + fb.addError("%v", err) + } action := µflows.WebServiceCallAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, ErrorHandlingType: convertErrorHandlingType(s.ErrorHandling), @@ -573,8 +578,12 @@ func (fb *flowBuilder) addCallWebServiceAction(s *ast.CallWebServiceStmt) model. // passed. The gate was green BECAUSE the fixture was broken — a valid // reference was the one input that triggered the defect, and nothing // tested one. - SendMappingID: model.ID(s.SendMappingID), - ReceiveMappingID: model.ID(s.ReceiveMappingID), + SendMappingID: model.ID(s.SendMappingID), + // The variable the export mapping maps FROM. Mendix stores it as + // MappingRequestHandling.MappingVariableName, and a send mapping without + // one cannot be written — refused above rather than written incomplete. + SendMappingVariable: s.SendMappingVariable, + ReceiveMappingID: model.ID(s.ReceiveMappingID), // The entity the receive mapping produces, which types the result // variable. Empty when unresolvable, and the writers keep VoidType. ResultEntity: resolveImportMappingEntity(fb.backend, s.ReceiveMappingID), @@ -592,6 +601,7 @@ func (fb *flowBuilder) addCallWebServiceAction(s *ast.CallWebServiceStmt) model. if s.Timeout != nil { action.TimeoutExpression = fb.exprToString(s.Timeout) } + action.Arguments = fb.webServiceArguments(s) activity := µflows.ActionActivity{ BaseActivity: microflows.BaseActivity{ @@ -622,6 +632,58 @@ func (fb *flowBuilder) addCallWebServiceAction(s *ast.CallWebServiceStmt) model. return activity.ID } +// webServiceArguments binds the statement's arguments to the stored +// ParameterPath each one needs. +// +// The path is derived from the OPERATION document rather than written by the +// author: Mendix stores +// `http%3A//www.example.com/:GetOrder|OrderId` where MDL says `OrderId`, and +// putting that in a script would fail every readability test the language is +// held to. The same move `send rest request` already makes — it stores each +// parameter under a qualified key and shows only the last segment. +// +// A path that cannot be derived is an ERROR, not a fallback. The other +// resolvers in this file fall back because their alternative is the value that +// ships today; there is no shipping value for a path that has never been +// written, and a fabricated one reproduces CE0178 with different text in it. +func (fb *flowBuilder) webServiceArguments(s *ast.CallWebServiceStmt) []microflows.WebServiceArgument { + if len(s.Arguments) == 0 { + return nil + } + element := resolveWebServiceOperationElement(fb.backend, s.ServiceID, s.OperationName) + if element == "" { + fb.addError("call web service %s: cannot resolve operation %s in the imported "+ + "service document, so the arguments have no parameter path to bind to.\n"+ + " Arguments need the consumed service to be present and to declare the "+ + "operation — check the name against `describe microflow` on an existing call, "+ + "or drop the argument list", + s.ServiceID, s.OperationName) + return nil + } + + out := make([]microflows.WebServiceArgument, 0, len(s.Arguments)) + for _, arg := range s.Arguments { + path := webServiceParameterPath(element, arg.Name) + if path == "" { + fb.addError("call web service %s: cannot build the parameter path for %q "+ + "from operation element %q — a name containing '%%' is refused because "+ + "Mendix's escaping of it is unverified", + s.ServiceID, arg.Name, element) + return nil + } + out = append(out, microflows.WebServiceArgument{ + Name: arg.Name, + Path: path, + Expression: fb.exprToString(arg.Value), + // Both reference mappings carry true, and Studio Pro's checkbox is + // ticked for a parameter that is bound at all — which an argument is, + // by being written. + Checked: true, + }) + } + return out +} + // resolveExternalActionReturnKind looks up the called OData action in the // consumed service's cached $metadata and returns the Mendix kind name // ("Boolean", "String", "Integer", "Long", "Decimal", "DateTime", "Binary", diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index bbd8c2913..40382dcad 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -1015,10 +1015,21 @@ func formatWebServiceCallAction(ctx *ExecContext, a *microflows.WebServiceCallAc // three resolvers that used to stand here actually did. parts := []string{prefix + "call web service " + formatWebServiceReference(string(a.ServiceID))} if a.OperationName != "" { - parts = append(parts, "operation "+formatWebServiceReference(a.OperationName)) + op := "operation " + formatWebServiceReference(a.OperationName) + // The arguments carry the stored ParameterPath, but only its last + // segment is spelled in MDL — the rest is rebuilt from the operation + // document on the way back in. + if args := formatWebServiceArguments(a.Arguments); args != "" { + op += " (" + args + ")" + } + parts = append(parts, op) } if a.SendMappingID != "" { - parts = append(parts, "send mapping "+formatWebServiceReference(string(a.SendMappingID))) + send := "send mapping " + formatWebServiceReference(string(a.SendMappingID)) + if a.SendMappingVariable != "" { + send += " from $" + a.SendMappingVariable + } + parts = append(parts, send) } if a.ReceiveMappingID != "" { parts = append(parts, "receive mapping "+formatWebServiceReference(string(a.ReceiveMappingID))) @@ -1029,6 +1040,27 @@ func formatWebServiceCallAction(ctx *ExecContext, a *microflows.WebServiceCallAc return strings.Join(parts, "\n") + ";" } +// formatWebServiceArguments renders the operation's argument list, or "" when +// there is nothing MDL can spell. +// +// An argument whose Name did not survive the read — a ParameterPath with no +// "|", which no reference document has — renders nothing at all, and the +// action falls back to the raw form rather than emitting a list that would +// write a different path back. +func formatWebServiceArguments(args []microflows.WebServiceArgument) string { + if len(args) == 0 { + return "" + } + parts := make([]string, 0, len(args)) + for _, arg := range args { + if arg.Name == "" { + return "" + } + parts = append(parts, arg.Name+" = "+arg.Expression) + } + return strings.Join(parts, ", ") +} + func formatWebServiceReference(ref string) string { if isBareQualifiedReference(ref) { return ref diff --git a/mdl/executor/cmd_microflows_format_action_test.go b/mdl/executor/cmd_microflows_format_action_test.go index 4e96c4d10..b7d229fa1 100644 --- a/mdl/executor/cmd_microflows_format_action_test.go +++ b/mdl/executor/cmd_microflows_format_action_test.go @@ -1360,3 +1360,67 @@ func TestFormatAction_WebServiceCallRaw(t *testing.T) { t.Fatalf("got %q", got) } } + +// TestFormatAction_WebServiceCallArgumentsAndSendMapping — DESCRIBE renders both +// request-body forms, and renders them so they parse back. +// +// The argument list shows only the parameter NAME; the stored ParameterPath +// ("http%3A//www.example.com/:GetOrder|OrderId") is rebuilt from the operation +// document on the way back in. Putting the path in the script would fail every +// readability test the language is held to — and `send rest request` already +// makes the same move, showing `code` where the model holds Mod.Svc.Op.code. +func TestFormatAction_WebServiceCallArgumentsAndSendMapping(t *testing.T) { + ctx, _ := newMockCtx(t, withBackend(&mock.MockBackend{IsConnectedFunc: func() bool { return true }}), + withHierarchy(mkHierarchy())) + + args := formatAction(ctx, µflows.WebServiceCallAction{ + ServiceID: "Clients.OrderSoapClient", + OperationName: "GetOrder", + Arguments: []microflows.WebServiceArgument{ + {Name: "OrderId", Path: "http%3A//www.example.com/:GetOrder|OrderId", Expression: "$Customer/OrderId", Checked: true}, + {Name: "Verbose", Path: "http%3A//www.example.com/:GetOrder|Verbose", Expression: "true", Checked: true}, + }, + ReceiveMappingID: "Clients.SoapOrdersImportMapping", + OutputVariable: "Orders", + }, nil, nil) + want := "$Orders = call web service Clients.OrderSoapClient\n" + + "operation GetOrder (OrderId = $Customer/OrderId, Verbose = true)\n" + + "receive mapping Clients.SoapOrdersImportMapping;" + if args != want { + t.Errorf("arguments form:\n got %q\nwant %q", args, want) + } + + send := formatAction(ctx, µflows.WebServiceCallAction{ + ServiceID: "Clients.OrderSoapClient", + OperationName: "SaveOrder", + SendMappingID: "Clients.SoapOrderExportMapping", + SendMappingVariable: "NewSaveOrder", + }, nil, nil) + wantSend := "call web service Clients.OrderSoapClient\n" + + "operation SaveOrder\n" + + "send mapping Clients.SoapOrderExportMapping from $NewSaveOrder;" + if send != wantSend { + t.Errorf("send mapping form:\n got %q\nwant %q", send, wantSend) + } +} + +// TestFormatAction_WebServiceCallArgumentWithoutAName renders no argument list +// at all rather than a partial one. +// +// A ParameterPath with no "|" yields no name on the way in — a shape no +// reference document carries — and emitting `operation X ( = expr)` would write +// a DIFFERENT path back. The action keeps the raw fallback instead, which is +// decided by the reader; this is the belt to that braces. +func TestFormatAction_WebServiceCallArgumentWithoutAName(t *testing.T) { + ctx, _ := newMockCtx(t, withBackend(&mock.MockBackend{IsConnectedFunc: func() bool { return true }}), + withHierarchy(mkHierarchy())) + + got := formatAction(ctx, µflows.WebServiceCallAction{ + ServiceID: "M.S", + OperationName: "Op", + Arguments: []microflows.WebServiceArgument{{Path: "no-separator", Expression: "1"}}, + }, nil, nil) + if strings.Contains(got, "(") { + t.Errorf("emitted a partial argument list: %q", got) + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index ef0bf91e4..d8e34ef69 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -352,6 +352,10 @@ func (v *microflowValidator) walkBody(body []ast.MicroflowStatement) { case *ast.RestCallStmt: // #922: `returns Module.Entity` must name a FileDocument specialization. v.checkRestFileDocumentResult(stmt) + case *ast.CallWebServiceStmt: + // MDL-SOAP01: a call stores ONE request body, so arguments and a send + // mapping are alternatives. Same function exec calls. + v.checkWebServiceRequestBody(stmt) case *ast.LoopStmt: // Check: @caption on a loop is silently dropped — Mendix for-loops // have no caption (Microflows$LoopedActivity has no Caption diff --git a/mdl/executor/validate_webservice_request_body.go b/mdl/executor/validate_webservice_request_body.go new file mode 100644 index 000000000..3f0bfe554 --- /dev/null +++ b/mdl/executor/validate_webservice_request_body.go @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation for a SOAP call's REQUEST BODY. +// +// `Microflows$CallWebServiceAction.RequestBodyHandling` is a polymorphic child +// and a call stores exactly ONE of them. The two a SOAP call uses are +// alternatives, not a pair: +// +// operation X (a = …) Microflows$SimpleRequestHandling +// send mapping M from $v Microflows$MappingRequestHandling +// +// The grammar admits both so this can name them. What it decides is decidable +// from the statement alone, so `mxcli check` reports it without a project — +// which is the point, because the alternative is what shipped before: both +// clauses accepted, neither written, and mxbuild reporting CE0369 several +// minutes later on a statement that mentions no "simple request body" at all. +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// checkWebServiceRequestBodyStmt reports what is wrong with a SOAP call's +// request-body clauses, or nil. +// +// This is the single decision: the executor calls it before building the +// action, and the check pass calls it through the microflow validator, so a +// script cannot pass one and fail the other. +func checkWebServiceRequestBodyStmt(s *ast.CallWebServiceStmt) error { + if s == nil || s.RawBSONBase64 != "" { + // A raw payload re-emits verbatim; its request body is whatever the + // bytes say, and no clause was parsed to contradict it. + return nil + } + + if len(s.Arguments) > 0 && s.SendMappingID != "" { + return fmt.Errorf( + "call web service %s: a call sends EITHER the operation's arguments OR an "+ + "export mapping, never both — Mendix stores one request body "+ + "(RequestBodyHandling), so one of the two would be silently dropped.\n"+ + " Keep `operation %s (…)` for a simple body, or `send mapping %s from $var` "+ + "for a mapped one", + s.ServiceID, s.OperationName, s.SendMappingID) + } + + if len(s.Arguments) > 0 && s.OperationName == "" { + return fmt.Errorf( + "call web service %s: arguments were given without an operation — the "+ + "parameter path is built from the operation's request body element, so "+ + "there is nothing to bind them to.\n"+ + " Add `operation ` before the argument list", + s.ServiceID) + } + + if s.SendMappingID != "" && s.SendMappingVariable == "" { + return fmt.Errorf( + "call web service %s: `send mapping %s` has no source variable — an export "+ + "mapping maps an OBJECT, and Mendix stores which one "+ + "(MappingRequestHandling.MappingVariableName), so mxcli cannot write the "+ + "mapping without it.\n"+ + " Write `send mapping %s from $YourVariable`", + s.ServiceID, s.SendMappingID, s.SendMappingID) + } + + for _, arg := range s.Arguments { + if arg.Name == "" { + return fmt.Errorf( + "call web service %s: an argument has no parameter name — each one binds a "+ + "named parameter of operation %s, e.g. `(OrderId = $Id)`", + s.ServiceID, s.OperationName) + } + } + return nil +} + +// checkWebServiceRequestBody surfaces checkWebServiceRequestBodyStmt as MDL-SOAP01. +func (v *microflowValidator) checkWebServiceRequestBody(stmt *ast.CallWebServiceStmt) { + err := checkWebServiceRequestBodyStmt(stmt) + if err == nil { + return + } + v.addViolation("MDL-SOAP01", linter.SeverityError, err.Error(), + "See `mxcli syntax soap` for the two request-body forms") +} diff --git a/mdl/executor/validate_webservice_request_body_test.go b/mdl/executor/validate_webservice_request_body_test.go new file mode 100644 index 000000000..5ff0188ce --- /dev/null +++ b/mdl/executor/validate_webservice_request_body_test.go @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// TestCheckWebServiceRequestBody_RefusesBothForms is the regression test for the +// silent drop this rule replaces. +// +// A call stores ONE RequestBodyHandling. Before this, both clauses parsed, the +// writer ignored the send mapping, and mxbuild reported CE0369 "Cannot use +// simple request body, as the operation's body is complex" — an error naming a +// simple request body on a statement that asked for a mapping. +func TestCheckWebServiceRequestBody_RefusesBothForms(t *testing.T) { + err := checkWebServiceRequestBodyStmt(&ast.CallWebServiceStmt{ + ServiceID: "Clients.OrderSoapClient", + OperationName: "SaveOrder", + Arguments: []ast.CallArgument{{Name: "OrderId", Value: nil}}, + SendMappingID: "Clients.SoapOrderExportMapping", + SendMappingVariable: "Order", + }) + if err == nil { + t.Fatal("a call asking for both arguments and a send mapping was accepted") + } + for _, want := range []string{"EITHER", "operation SaveOrder", "send mapping"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("message does not name %q: %v", want, err) + } + } +} + +// TestCheckWebServiceRequestBody_SendMappingNeedsAVariable — an export mapping +// maps an OBJECT, and Mendix stores which one. mxcli cannot write the mapping +// without it, so the clause is refused rather than written incomplete. +func TestCheckWebServiceRequestBody_SendMappingNeedsAVariable(t *testing.T) { + err := checkWebServiceRequestBodyStmt(&ast.CallWebServiceStmt{ + ServiceID: "Clients.OrderSoapClient", + OperationName: "SaveOrder", + SendMappingID: "Clients.SoapOrderExportMapping", + }) + if err == nil { + t.Fatal("`send mapping X` with no source variable was accepted") + } + if !strings.Contains(err.Error(), "from $") { + t.Errorf("message does not show the fix: %v", err) + } +} + +// TestCheckWebServiceRequestBody_ArgumentsNeedAnOperation — the parameter path +// is built from the operation's request body element, so arguments without an +// operation have nothing to bind to. +func TestCheckWebServiceRequestBody_ArgumentsNeedAnOperation(t *testing.T) { + if err := checkWebServiceRequestBodyStmt(&ast.CallWebServiceStmt{ + ServiceID: "Clients.OrderSoapClient", + Arguments: []ast.CallArgument{{Name: "OrderId"}}, + }); err == nil { + t.Fatal("arguments without an operation were accepted") + } +} + +// TestCheckWebServiceRequestBody_Accepts covers every shape that is valid, +// including the ones that were valid before this rule existed — a rule that +// refuses working scripts is worse than the drop it replaces. +func TestCheckWebServiceRequestBody_Accepts(t *testing.T) { + for _, tc := range []struct { + name string + stmt *ast.CallWebServiceStmt + }{ + {"arguments only", &ast.CallWebServiceStmt{ + ServiceID: "M.S", OperationName: "Op", + Arguments: []ast.CallArgument{{Name: "OrderId"}}, + }}, + {"send mapping with its variable", &ast.CallWebServiceStmt{ + ServiceID: "M.S", OperationName: "Op", + SendMappingID: "M.Export", SendMappingVariable: "Order", + }}, + {"neither — today's argument-less call", &ast.CallWebServiceStmt{ + ServiceID: "M.S", OperationName: "Op", ReceiveMappingID: "M.Import", + }}, + // A raw payload re-emits verbatim, so its request body is whatever the + // bytes say and no clause was parsed to contradict it. + {"raw payload with clauses that would otherwise clash", &ast.CallWebServiceStmt{ + RawBSONBase64: "AQID", + Arguments: []ast.CallArgument{{Name: "OrderId"}}, + SendMappingID: "M.Export", + }}, + {"nil statement", nil}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := checkWebServiceRequestBodyStmt(tc.stmt); err != nil { + t.Errorf("refused a valid call: %v", err) + } + }) + } +} diff --git a/mdl/executor/webservice_names.go b/mdl/executor/webservice_names.go index b97a9a425..481393eff 100644 --- a/mdl/executor/webservice_names.go +++ b/mdl/executor/webservice_names.go @@ -65,12 +65,22 @@ const importedServiceType = "WebServices$ImportedServiceImpl" // fallback in the rare two-modules-same-name case and never picks the wrong // service. func resolveWebServiceName(b backend.FullBackend, qualifiedName, operationName string) string { - if b == nil || qualifiedName == "" { + matched := findImportedServiceDoc(b, qualifiedName) + if matched == nil { return "" } + return serviceNameFromImportedService(matched, operationName) +} + +// findImportedServiceDoc returns the contents of the imported service document +// named by qualifiedName, or nil when it cannot be identified unambiguously. +func findImportedServiceDoc(b backend.FullBackend, qualifiedName string) []byte { + if b == nil || qualifiedName == "" { + return nil + } units, err := b.ListRawUnitsByType(importedServiceType) if err != nil || len(units) == 0 { - return "" + return nil } _, bare, ok := strings.Cut(qualifiedName, ".") if !ok || bare == "" { @@ -86,14 +96,11 @@ func resolveWebServiceName(b backend.FullBackend, qualifiedName, operationName s continue } if matched != nil { - return "" // ambiguous — two documents of this name + return nil // ambiguous — two documents of this name } matched = unit.Contents } - if matched == nil { - return "" - } - return serviceNameFromImportedService(matched, operationName) + return matched } // serviceNameFromImportedService reads Description.Services[] and returns the @@ -131,6 +138,89 @@ func serviceNameFromImportedService(contents []byte, operationName string) strin return "" } +// resolveWebServiceOperationElement returns the operation's stored +// RequestBodyElementName — `"http://www.example.com/:GetOrder"` for +// ako/TestApp's GetOrder — which is the prefix of every argument's ParameterPath. +// +// "" when it cannot be established, and the caller then REFUSES to write the +// arguments rather than inventing a path. That is stricter than the other +// resolvers here, which fall back: falling back is safe when the alternative is +// the value that ships today, and there is no such value for a path that has +// never been written. +func resolveWebServiceOperationElement(b backend.FullBackend, qualifiedName, operationName string) string { + if operationName == "" { + return "" + } + matched := findImportedServiceDoc(b, qualifiedName) + if matched == nil { + return "" + } + var doc map[string]any + if err := bson.Unmarshal(matched, &doc); err != nil { + return "" + } + for _, svc := range typedArrayElements(docLookup(doc["Description"], "Services")) { + for _, op := range typedArrayElements(docLookup(svc, "Operations")) { + if name, _ := docLookup(op, "Name").(string); !strings.EqualFold(name, operationName) { + continue + } + element, _ := docLookup(op, "RequestBodyElementName").(string) + return element + } + } + return "" +} + +// webServiceParameterPath builds the stored ParameterPath for one argument. +// +// Measured on ako/TestApp (11.14.0): the operation element +// `http://www.example.com/:GetOrder` and the parameter `OrderId` are stored as +// +// http%3A//www.example.com/:GetOrder|OrderId +// +// so the element's namespace and local name keep the `:` BETWEEN them and the +// `|` before the parameter, while a `:` INSIDE a segment is percent-encoded. The +// element name splits on its LAST colon, since the namespace is a URI and +// carries colons of its own. +// +// "" when the path cannot be built, which the caller turns into a refusal. +func webServiceParameterPath(operationElement, parameterName string) string { + if operationElement == "" || parameterName == "" { + return "" + } + namespace, local := "", operationElement + if i := strings.LastIndex(operationElement, ":"); i >= 0 { + namespace, local = operationElement[:i], operationElement[i+1:] + } + ns, okNS := escapeParameterPathSegment(namespace) + lo, okLO := escapeParameterPathSegment(local) + pn, okPN := escapeParameterPathSegment(parameterName) + if !okNS || !okLO || !okPN { + return "" + } + if namespace == "" { + return lo + "|" + pn + } + return ns + ":" + lo + "|" + pn +} + +// escapeParameterPathSegment percent-encodes the two characters that would +// otherwise be read as path structure. +// +// Only `:` has a reference document behind it; `|` is escaped by the same +// reasoning and has never been observed in a namespace. A segment already +// containing a `%` is REFUSED (ok=false) rather than encoded or passed through: +// whether Mendix escapes it as %25 is unmeasured, and both answers produce a +// path that silently addresses the wrong parameter. +func escapeParameterPathSegment(s string) (string, bool) { + if strings.Contains(s, "%") { + return "", false + } + s = strings.ReplaceAll(s, ":", "%3A") + s = strings.ReplaceAll(s, "|", "%7C") + return s, true +} + // serviceDeclaresOperation reports whether a WebServices$ServiceInfoImpl lists an // operation of this name. func serviceDeclaresOperation(svc any, operationName string) bool { diff --git a/mdl/executor/webservice_names_test.go b/mdl/executor/webservice_names_test.go index 67dea5de8..a0ed560c4 100644 --- a/mdl/executor/webservice_names_test.go +++ b/mdl/executor/webservice_names_test.go @@ -240,3 +240,95 @@ func TestResolveImportMappingEntity_UnresolvableIsEmpty(t *testing.T) { t.Errorf("nil backend = %q, want \"\"", got) } } + +// operationDoc builds an imported service whose single service declares +// operations with their RequestBodyElementName — the shape the ParameterPath +// derivation reads. +func operationDoc(t *testing.T, docName, serviceName string, ops map[string]string) []byte { + t.Helper() + opArr := bson.A{int32(2)} + for name, element := range ops { + opArr = append(opArr, bson.M{ + "$Type": "WebServices$OperationInfoImpl", + "Name": name, "RequestBodyElementName": element, + }) + } + out, err := bson.Marshal(bson.M{ + "$Type": importedServiceType, "Name": docName, + "Description": bson.M{"Services": bson.A{int32(2), bson.M{ + "$Type": "WebServices$ServiceInfoImpl", "Name": serviceName, "Operations": opArr, + }}}, + }) + if err != nil { + t.Fatalf("marshal: %v", err) + } + return out +} + +// TestWebServiceParameterPath pins the escaping character for character. +// +// Measured on ako/TestApp (11.14.0): operation element +// "http://www.example.com/:GetOrder" and parameter "OrderId" are stored as +// "http%3A//www.example.com/:GetOrder|OrderId". Note what is NOT escaped — the +// slashes, and the colon BETWEEN namespace and local name — because a plausible +// wrong escaping is exactly what mxbuild accepts and Studio Pro does not. +func TestWebServiceParameterPath(t *testing.T) { + got := webServiceParameterPath("http://www.example.com/:GetOrder", "OrderId") + if want := "http%3A//www.example.com/:GetOrder|OrderId"; got != want { + t.Errorf("webServiceParameterPath = %q, want %q", got, want) + } + // An element with no namespace at all keeps the bare local name. + if got := webServiceParameterPath("GetOrder", "OrderId"); got != "GetOrder|OrderId" { + t.Errorf("unqualified element = %q, want GetOrder|OrderId", got) + } + // A "|" inside a segment would otherwise be read as the separator. + if got := webServiceParameterPath("urn:a|b:Op", "P"); got != "urn%3Aa%7Cb:Op|P" { + t.Errorf("pipe not escaped: %q", got) + } +} + +// TestWebServiceParameterPath_RefusesUnverifiableEscaping — a segment already +// containing "%" is refused rather than encoded or passed through. Whether +// Mendix writes %25 there is unmeasured, and both answers produce a path that +// silently addresses the wrong parameter. +func TestWebServiceParameterPath_RefusesUnverifiableEscaping(t *testing.T) { + for _, tc := range []struct{ element, name string }{ + {"http://x/%y:Op", "P"}, + {"http://x/:Op", "P%1"}, + {"", "P"}, + {"http://x/:Op", ""}, + } { + if got := webServiceParameterPath(tc.element, tc.name); got != "" { + t.Errorf("webServiceParameterPath(%q, %q) = %q, want \"\"", tc.element, tc.name, got) + } + } +} + +// TestResolveWebServiceOperationElement reads the prefix every argument's path +// is built from, off the operation rather than out of the WSDL text. +func TestResolveWebServiceOperationElement(t *testing.T) { + b := backendWithUnits(operationDoc(t, "OrderSoapClient", "OrdersWS", map[string]string{ + "GetOrder": "http://www.example.com/:GetOrder", + "SaveOrder": "http://www.example.com/:SaveOrder", + })) + + if got := resolveWebServiceOperationElement(b, "Clients.OrderSoapClient", "GetOrder"); got != "http://www.example.com/:GetOrder" { + t.Errorf("= %q", got) + } + // Unresolvable every way returns "", and the caller REFUSES rather than + // falling back — there is no shipping value for a path never written. + for _, tc := range []struct{ name, qn, op string }{ + {"no such operation", "Clients.OrderSoapClient", "Nope"}, + {"no such document", "Clients.Missing", "GetOrder"}, + {"no operation named", "Clients.OrderSoapClient", ""}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := resolveWebServiceOperationElement(b, tc.qn, tc.op); got != "" { + t.Errorf("= %q, want \"\"", got) + } + }) + } + if got := resolveWebServiceOperationElement(nil, "Clients.OrderSoapClient", "GetOrder"); got != "" { + t.Errorf("nil backend = %q", got) + } +} diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 8e2be61c4..c0830f755 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -474,12 +474,24 @@ callJavaScriptActionStatement ; // Legacy SOAP call. +// +// The request body is EITHER the operation's arguments OR an export mapping — +// Mendix stores one RequestBodyHandling, not two — so writing both is refused +// by `mxcli check`. The grammar admits both so the refusal can name them. +// +// Arguments parenthesise on OPERATION, matching CALL EXTERNAL ACTION: an OData +// action and a SOAP operation are the same shape of thing, and `operation X` is +// the callee here (the statement's own target is the service). +// +// SEND MAPPING … FROM $var mirrors REST's `body mapping … from $var`. FROM +// cannot be swallowed by the preceding qualifiedName — that rule only continues +// across a DOT — which is why the same shape already works there. callWebServiceStatement : (VARIABLE EQUALS)? CALL WEB SERVICE (RAW STRING_LITERAL | webServiceReference - (OPERATION webServiceReference)? - (SEND MAPPING webServiceReference)? + (OPERATION webServiceReference (LPAREN callArgumentList? RPAREN)?)? + (SEND MAPPING webServiceReference (FROM VARIABLE)?)? (RECEIVE MAPPING webServiceReference)? (TIMEOUT expression)?) onErrorClause? diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index c299ed790..9d12439bf 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -530,8 +530,15 @@ func buildCallWebServiceStatement(ctx parser.ICallWebServiceStatementContext) *a callCtx := ctx.(*parser.CallWebServiceStatementContext) stmt := &ast.CallWebServiceStmt{} - if v := callCtx.VARIABLE(); v != nil { - stmt.OutputVariable = strings.TrimPrefix(v.GetText(), "$") + // Two VARIABLE positions now: the output variable before EQUALS, and the + // send mapping's source after FROM. Walk them positionally against the + // tokens that gate each, as the webServiceReference walk below does — with + // no output variable, the send variable IS the first one. + vars := callCtx.AllVARIABLE() + varIdx := 0 + if callCtx.EQUALS() != nil && len(vars) > varIdx { + stmt.OutputVariable = strings.TrimPrefix(vars[varIdx].GetText(), "$") + varIdx++ } if callCtx.RAW() != nil { @@ -556,10 +563,16 @@ func buildCallWebServiceStatement(ctx parser.ICallWebServiceStatementContext) *a if callCtx.OPERATION() != nil && len(refs) > idx { stmt.OperationName = webServiceReferenceText(refs[idx]) idx++ + if argList := callCtx.CallArgumentList(); argList != nil { + stmt.Arguments = buildCallArgumentList(argList) + } } if callCtx.SEND() != nil && len(refs) > idx { stmt.SendMappingID = webServiceReferenceText(refs[idx]) idx++ + if callCtx.FROM() != nil && len(vars) > varIdx { + stmt.SendMappingVariable = strings.TrimPrefix(vars[varIdx].GetText(), "$") + } } if callCtx.RECEIVE() != nil && len(refs) > idx { stmt.ReceiveMappingID = webServiceReferenceText(refs[idx]) diff --git a/mdl/visitor/visitor_webservice_test.go b/mdl/visitor/visitor_webservice_test.go index 36bc70f07..6d117b906 100644 --- a/mdl/visitor/visitor_webservice_test.go +++ b/mdl/visitor/visitor_webservice_test.go @@ -84,3 +84,64 @@ func TestCallWebServiceRawStatement(t *testing.T) { t.Errorf("raw statement should not set structured refs: %#v", call) } } + +// TestCallWebServiceArguments — the operation's arguments use callArgumentList, +// the same `(Name = value)` form every other call statement in MDL uses, so the +// visitor reuses the same builder. +func TestCallWebServiceArguments(t *testing.T) { + stmt := firstStatement(t, `$Order = call web service Clients.OrderSoapClient +operation GetOrder (OrderId = $Customer/OrderId, Verbose = true) +receive mapping Clients.SoapOrdersImportMapping;`) + + call := stmt.(*ast.CallWebServiceStmt) + if call.OperationName != "GetOrder" { + t.Fatalf("OperationName = %q", call.OperationName) + } + if len(call.Arguments) != 2 { + t.Fatalf("read %d arguments, want 2: %#v", len(call.Arguments), call.Arguments) + } + if call.Arguments[0].Name != "OrderId" || call.Arguments[1].Name != "Verbose" { + t.Errorf("argument names = %q, %q", call.Arguments[0].Name, call.Arguments[1].Name) + } + if call.Arguments[0].Value == nil { + t.Error("argument expression not built") + } +} + +// TestCallWebServiceSendMappingVariable — `from $var` names the object the +// export mapping maps, which Mendix stores as MappingVariableName. +// +// The output variable and this one are both VARIABLE tokens in the same rule, +// so the visitor walks them positionally against EQUALS and FROM. The second +// case below is the one that breaks a naive "first VARIABLE is the output" +// reading: there is no output variable, so the FIRST variable in the statement +// is the send mapping's. +func TestCallWebServiceSendMappingVariable(t *testing.T) { + withOutput := firstStatement(t, `$Ok = call web service Clients.OrderSoapClient +operation SaveOrder +send mapping Clients.SoapOrderExportMapping from $NewSaveOrder;`).(*ast.CallWebServiceStmt) + if withOutput.OutputVariable != "Ok" || withOutput.SendMappingVariable != "NewSaveOrder" { + t.Errorf("output = %q, send variable = %q", withOutput.OutputVariable, withOutput.SendMappingVariable) + } + + noOutput := firstStatement(t, `call web service Clients.OrderSoapClient +operation SaveOrder +send mapping Clients.SoapOrderExportMapping from $NewSaveOrder;`).(*ast.CallWebServiceStmt) + if noOutput.OutputVariable != "" { + t.Errorf("OutputVariable = %q, want empty", noOutput.OutputVariable) + } + if noOutput.SendMappingVariable != "NewSaveOrder" { + t.Errorf("SendMappingVariable = %q, want NewSaveOrder", noOutput.SendMappingVariable) + } +} + +// TestCallWebServiceWithoutNewClauses — every statement that parsed before still +// parses, with both new fields empty. +func TestCallWebServiceWithoutNewClauses(t *testing.T) { + call := firstStatement(t, `$Root = call web service SampleSOAP.OrderService +operation FetchSampleItems +receive mapping SampleSOAP.OrderResponse;`).(*ast.CallWebServiceStmt) + if len(call.Arguments) != 0 || call.SendMappingVariable != "" { + t.Errorf("new fields set on an old-shape statement: %#v", call) + } +} diff --git a/sdk/microflows/microflows_actions.go b/sdk/microflows/microflows_actions.go index b2a127584..fa8a82eb5 100644 --- a/sdk/microflows/microflows_actions.go +++ b/sdk/microflows/microflows_actions.go @@ -786,13 +786,28 @@ type WebServiceCallAction struct { // operation within this one. Resolved from the imported service document by // the executor (see resolveWebServiceName); empty means it could not be // established, and the writers fall back to deriving it from ServiceID. - ServiceName string `json:"serviceName,omitempty"` - OperationName string `json:"operationName,omitempty"` - SendMappingID model.ID `json:"sendMappingId,omitempty"` - ReceiveMappingID model.ID `json:"receiveMappingId,omitempty"` - OutputVariable string `json:"outputVariable,omitempty"` - UseReturnVariable bool `json:"useReturnVariable"` - TimeoutExpression string `json:"timeoutExpression,omitempty"` + ServiceName string `json:"serviceName,omitempty"` + OperationName string `json:"operationName,omitempty"` + // Arguments binds the operation's parameters — Mendix's + // Microflows$SimpleRequestHandling. Mutually exclusive with SendMappingID: + // a call stores ONE RequestBodyHandling, and the executor refuses a + // statement that asks for both. + Arguments []WebServiceArgument `json:"arguments,omitempty"` + SendMappingID model.ID `json:"sendMappingId,omitempty"` + // SendMappingVariable is the variable the send mapping maps FROM + // (MappingRequestHandling.MappingVariableName). An export mapping always + // maps an object, so a send mapping without it is incomplete. + SendMappingVariable string `json:"sendMappingVariable,omitempty"` + // SendMappingContentType is the stored ContentType of a send mapping, + // carried through a rewrite rather than normalised. Studio Pro wrote "Json" + // on the one reference document available (ako/TestApp Clients.SaveOrder), + // which is surprising on an XML protocol and is why this is preserved + // rather than derived. Empty means "write the default". + SendMappingContentType string `json:"sendMappingContentType,omitempty"` + ReceiveMappingID model.ID `json:"receiveMappingId,omitempty"` + OutputVariable string `json:"outputVariable,omitempty"` + UseReturnVariable bool `json:"useReturnVariable"` + TimeoutExpression string `json:"timeoutExpression,omitempty"` // ResultEntity is the qualified entity the RECEIVE mapping produces, which // Mendix stores as the call's result VariableType. Resolved from the mapping // document by the executor; empty means it could not be established and the @@ -802,6 +817,25 @@ type WebServiceCallAction struct { func (WebServiceCallAction) isMicroflowAction() {} +// WebServiceArgument binds one SOAP operation parameter to an expression — +// a Microflows$WebServiceOperationSimpleParameterMapping. +// +// Name is the bare parameter name as MDL spells it ("OrderId"). Path is the +// stored ParameterPath, which is the operation's escaped RequestBodyElementName +// plus "|" plus the name ("http%3A//www.example.com/:GetOrder|OrderId"). The +// executor derives Path from the operation document; a Path that could not be +// established stays empty and the call is refused rather than written with a +// fabricated one, since a wrong path reproduces CE0178 with different text in it. +type WebServiceArgument struct { + Name string `json:"name"` + Path string `json:"path,omitempty"` + Expression string `json:"expression,omitempty"` + // Checked mirrors the stored IsChecked. Both reference mappings carry true + // and no false has been observed, so it defaults to true on a fresh write + // and is preserved on a rewrite rather than normalised. + Checked bool `json:"checked"` +} + // RestCallAction calls a REST service. type RestCallAction struct { model.BaseElement diff --git a/sdk/mpr/parser_microflow_actions.go b/sdk/mpr/parser_microflow_actions.go index 914a6b74b..b41ff8a5f 100644 --- a/sdk/mpr/parser_microflow_actions.go +++ b/sdk/mpr/parser_microflow_actions.go @@ -3,6 +3,8 @@ package mpr import ( + "strings" + "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/microflows" @@ -472,11 +474,15 @@ func parseWebServiceCallAction(raw map[string]any) *microflows.WebServiceCallAct action.ReceiveMappingID = model.ID(extractString(call["ReturnValueMapping"])) } } + // RequestHandling / ExportMappingCall is a shape no reference document + // carries — the real key is RequestBodyHandling, read below — so this never + // populated SendMappingID from a real project. if requestHandling := extractBsonMap(raw["RequestHandling"]); requestHandling != nil { if call := extractBsonMap(requestHandling["ExportMappingCall"]); call != nil { action.SendMappingID = model.ID(extractString(call["Mapping"])) } } + parseWebServiceRequestBody(raw, action) if webServiceActionRequiresRawBSON(raw) { if rawBSON, err := bson.Marshal(raw); err == nil { action.RawBSON = rawBSON @@ -486,26 +492,206 @@ func parseWebServiceCallAction(raw map[string]any) *microflows.WebServiceCallAct return action } +// parseWebServiceRequestBody reads a SOAP call's RequestBodyHandling back into +// the semantic model. Mirrors modelsdkbackend.readWebServiceRequestBody. +// +// Dispatched on $Type, never on which fields are present: MappingRequestHandling +// and SimpleRequestHandling differ in arity, so assigning whichever keys turn up +// would quietly turn one into the other. +func parseWebServiceRequestBody(raw map[string]any, action *microflows.WebServiceCallAction) { + body := extractBsonMap(raw["RequestBodyHandling"]) + if body == nil { + return + } + switch extractString(body["$Type"]) { + case "Microflows$MappingRequestHandling": + // STORAGE NAMES: MappingId / MappingVariableName — not gen's Mapping / + // MappingArgumentVariableName, both of which its key audit lists as wrong. + action.SendMappingID = model.ID(extractString(body["MappingId"])) + action.SendMappingVariable = extractString(body["MappingVariableName"]) + action.SendMappingContentType = extractString(body["ContentType"]) + case "Microflows$SimpleRequestHandling": + for _, el := range extractBsonArray(body["ParameterMappings"]) { + pm := extractBsonMap(el) + if pm == nil || extractString(pm["$Type"]) != "Microflows$WebServiceOperationSimpleParameterMapping" { + // An advanced (per-parameter export mapping) entry, which MDL + // cannot author. The raw fallback carries the action; reading + // half of it here would be worse than reading none. + continue + } + path := extractString(pm["ParameterPath"]) + name := "" + if i := strings.LastIndex(path, "|"); i >= 0 { + name = path[i+1:] + } + action.Arguments = append(action.Arguments, microflows.WebServiceArgument{ + Name: name, + Path: path, + Expression: extractString(pm["Argument"]), + // Absent reads as true: both reference mappings carry true, and + // a bound parameter is one Studio Pro has ticked. + Checked: extractBool(pm["IsChecked"], true), + }) + } + } +} + +// webServiceActionRequiresRawBSON reports whether the structured describe form +// would fail to reproduce this action, in which case the renderer falls back to +// `call web service raw ''`. Mirrors +// modelsdkbackend.webServiceActionRequiresRawBSON decision for decision — see the +// comment there for why the six boilerplate keys are admitted only AT the value +// mxcli writes rather than by name. func webServiceActionRequiresRawBSON(raw map[string]any) bool { - supported := map[string]bool{ + represented := map[string]bool{ "$ID": true, "$Type": true, "ErrorHandlingType": true, "ImportedService": true, "OperationName": true, "TimeOutExpression": true, - "UseRequestTimeOut": true, - "NewResultHandling": true, "RequestHandling": true, + // Re-read from the imported service document on write (the CE0386 fix), + // so DESCRIBE need not carry it. + "ServiceName": true, } - for key := range raw { - if !supported[key] { + for key, value := range raw { + if represented[key] { + continue + } + ok, known := webServiceFixedValueIsDefault(key, value) + if !known || !ok { return true } } return false } +// webServiceFixedValueIsDefault reports whether one of the keys mxcli writes at a +// FIXED value currently holds it. known is false for a key it does not judge. +func webServiceFixedValueIsDefault(key string, value any) (ok, known bool) { + switch key { + case "IsValidationRequired": + return !extractBool(value, true), true + case "UseRequestTimeOut": + return extractBool(value, false), true + case "RequestProxyType": + return extractString(value) == "DefaultProxy", true + case "ProxyConfiguration": + return value == nil, true + case "HttpConfiguration": + return isDefaultWebServiceHTTPConfig(extractBsonMap(value)), true + case "RequestHeaderHandling": + return isEmptySimpleRequestHandling(extractBsonMap(value)), true + case "RequestBodyHandling": + return webServiceRequestBodyIsRepresentable(extractBsonMap(value)), true + case "NewResultHandling": + return webServiceResultHandlingIsRepresentable(extractBsonMap(value)), true + } + return false, false +} + +// webServiceResultHandlingIsRepresentable reports whether a call's result +// handling is one the writer reproduces exactly. See the comment on the +// modelsdk twin for the two ako/TestApp calls that prove it cannot be admitted +// by name: a BooleanType result with no mapping, and Range.SingleObject false. +func webServiceResultHandlingIsRepresentable(doc map[string]any) bool { + if doc == nil || extractString(doc["$Type"]) != "Microflows$ResultHandling" { + return false + } + bound := extractString(doc["ResultVariableName"]) != "" + if extractBool(doc["Bind"], !bound) != bound { + return false + } + vt := extractBsonMap(doc["VariableType"]) + if vt == nil { + return false + } + imc := extractBsonMap(doc["ImportMappingCall"]) + if imc == nil { + return extractString(vt["$Type"]) == "DataTypes$VoidType" + } + if extractString(vt["$Type"]) != "DataTypes$ObjectType" { + return false + } + if extractString(imc["$Type"]) != "Microflows$ImportMappingCall" || + extractString(imc["Commit"]) != "YesWithoutEvents" || + extractString(imc["ContentType"]) != "Xml" || + extractString(imc["ObjectHandlingBackup"]) != "Create" || + extractString(imc["ParameterVariableName"]) != "" || + extractString(imc["ReturnValueMapping"]) == "" || + extractBool(imc["ForceSingleOccurrence"], true) { + return false + } + rng := extractBsonMap(imc["Range"]) + return rng != nil && + extractString(rng["$Type"]) == "Microflows$ConstantRange" && + extractBool(rng["SingleObject"], false) +} + +// isDefaultWebServiceHTTPConfig reports whether an HttpConfiguration is the one a +// SOAP call gets when nothing is configured — the only one mxcli writes. +func isDefaultWebServiceHTTPConfig(doc map[string]any) bool { + if doc == nil || extractString(doc["$Type"]) != "Microflows$HttpConfiguration" { + return false + } + for _, key := range []string{"ClientCertificate", "CustomLocation", + "HttpAuthenticationPassword", "HttpAuthenticationUserName"} { + if extractString(doc[key]) != "" { + return false + } + } + if doc["CustomLocationTemplate"] != nil { + return false + } + if extractString(doc["HttpMethod"]) != "Post" { + return false + } + if extractBool(doc["OverrideLocation"], true) || extractBool(doc["UseHttpAuthentication"], true) { + return false + } + return len(extractBsonArray(doc["HttpHeaderEntries"])) == 0 +} + +// isEmptySimpleRequestHandling reports whether a request handling is the bare +// Simple form — no parameter mappings — which is all mxcli writes for headers. +func isEmptySimpleRequestHandling(doc map[string]any) bool { + return doc != nil && + extractString(doc["$Type"]) == "Microflows$SimpleRequestHandling" && + extractString(doc["NullValueOption"]) == "LeaveOutElement" && + len(extractBsonArray(doc["ParameterMappings"])) == 0 +} + +// webServiceRequestBodyIsRepresentable reports whether a RequestBodyHandling is +// one MDL can spell: an export mapping, or simple parameter mappings whose names +// survive the round trip. +func webServiceRequestBodyIsRepresentable(doc map[string]any) bool { + if doc == nil { + return false + } + switch extractString(doc["$Type"]) { + case "Microflows$MappingRequestHandling": + return extractString(doc["MappingId"]) != "" && extractString(doc["MappingVariableName"]) != "" + case "Microflows$SimpleRequestHandling": + if extractString(doc["NullValueOption"]) != "LeaveOutElement" { + return false + } + for _, el := range extractBsonArray(doc["ParameterMappings"]) { + pm := extractBsonMap(el) + if pm == nil || + extractString(pm["$Type"]) != "Microflows$WebServiceOperationSimpleParameterMapping" || + extractString(pm["ParameterName"]) != "" { + return false + } + if !strings.Contains(extractString(pm["ParameterPath"]), "|") { + return false + } + } + return true + } + return false +} + func parseWebServiceCallActionFromD(raw primitive.D) *microflows.WebServiceCallAction { action := parseWebServiceCallAction(raw.Map()) if rawBSON, err := bson.Marshal(raw); err == nil { diff --git a/sdk/mpr/writer_microflow_actions.go b/sdk/mpr/writer_microflow_actions.go index c46223142..2299c5569 100644 --- a/sdk/mpr/writer_microflow_actions.go +++ b/sdk/mpr/writer_microflow_actions.go @@ -868,18 +868,22 @@ func serializeWebServiceCallAction(a *microflows.WebServiceCallAction) bson.D { doc = append(doc, bson.E{Key: "OperationName", Value: a.OperationName}) doc = append(doc, bson.E{Key: "ProxyConfiguration", Value: nil}) - // RequestBodyHandling: always SimpleRequestHandling. Mendix$AdvancedRequestHandling - // (used when a send mapping is configured) requires a Studio Pro-generated example - // to determine the correct type storage name; use the raw BSON escape hatch for - // complex SOAP operations that need a send mapping until that is resolved. - doc = append(doc, bson.E{Key: "RequestBodyHandling", Value: bson.D{ - {Key: "$ID", Value: idToBsonBinary(GenerateID())}, - {Key: "$Type", Value: "Microflows$SimpleRequestHandling"}, - {Key: "NullValueOption", Value: "LeaveOutElement"}, - {Key: "ParameterMappings", Value: bson.A{int32(2)}}, - }}) - - // RequestHeaderHandling is always SimpleRequestHandling. + // RequestBodyHandling holds EITHER the operation's arguments or an export + // mapping — one polymorphic child, never both, which is why the executor + // refuses a statement asking for each (MDL-SOAP01). + // + // This used to be an unconditional empty SimpleRequestHandling. Both halves + // of that were wrong against ako/TestApp: an operation taking parameters + // needs them (CE0178 "Body parameter mapping needs to be refreshed"), and a + // send mapping is a Microflows$MappingRequestHandling — NOT the + // Mendix$AdvancedRequestHandling this comment used to name, a type that + // appears in none of the three reference documents. Writing Simple regardless + // dropped the mapping silently and gave CE0369 "Cannot use simple request + // body, as the operation's body is complex". + doc = append(doc, bson.E{Key: "RequestBodyHandling", Value: webServiceRequestBody(a)}) + + // RequestHeaderHandling is always SimpleRequestHandling: MDL cannot author + // SOAP headers, and all three reference calls carry the bare form. doc = append(doc, bson.E{Key: "RequestHeaderHandling", Value: bson.D{ {Key: "$ID", Value: idToBsonBinary(GenerateID())}, {Key: "$Type", Value: "Microflows$SimpleRequestHandling"}, @@ -896,6 +900,55 @@ func serializeWebServiceCallAction(a *microflows.WebServiceCallAction) bson.D { return doc } +// webServiceRequestBody builds a SOAP call's RequestBodyHandling — the arguments +// form or the export-mapping form. Mirrors +// modelsdkbackend.webServiceRequestBodyToGen key for key. +func webServiceRequestBody(a *microflows.WebServiceCallAction) bson.D { + if a.SendMappingID != "" { + // MappingId / MappingVariableName are the STORAGE names. modelsdk/gen + // binds the same two properties as Mapping and + // MappingArgumentVariableName (its key audit lists both), and a document + // written under those is one mxbuild tolerates and Studio Pro cannot + // open — so the legacy writer, which names keys directly, is the easier + // of the two engines to get right here. + contentType := a.SendMappingContentType + if contentType == "" { + // What Studio Pro wrote on the one reference document + // (ako/TestApp Clients.SaveOrder) — surprising on an XML protocol, + // hence preserved on a rewrite rather than derived. + contentType = "Json" + } + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "Microflows$MappingRequestHandling"}, + {Key: "ContentType", Value: contentType}, + {Key: "MappingId", Value: string(a.SendMappingID)}, + {Key: "MappingVariableName", Value: a.SendMappingVariable}, + } + } + + // Marker 2, measured on all three reference calls. + mappings := bson.A{int32(2)} + for _, arg := range a.Arguments { + mappings = append(mappings, bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "Microflows$WebServiceOperationSimpleParameterMapping"}, + {Key: "Argument", Value: arg.Expression}, + {Key: "IsChecked", Value: arg.Checked}, + // "" in both reference mappings; what fills it is unmeasured, so it + // is written empty rather than guessed at from the argument's name. + {Key: "ParameterName", Value: ""}, + {Key: "ParameterPath", Value: arg.Path}, + }) + } + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "Microflows$SimpleRequestHandling"}, + {Key: "NullValueOption", Value: "LeaveOutElement"}, + {Key: "ParameterMappings", Value: mappings}, + } +} + // serializeRestOperationCallAction serializes a Microflows$RestOperationCallAction to BSON. // Note: RestOperationCallAction does not support custom ErrorHandlingType (CE6035). func serializeRestOperationCallAction(a *microflows.RestOperationCallAction) bson.D { diff --git a/sdk/mpr/writer_webservice_body_test.go b/sdk/mpr/writer_webservice_body_test.go new file mode 100644 index 000000000..4e95a5a43 --- /dev/null +++ b/sdk/mpr/writer_webservice_body_test.go @@ -0,0 +1,275 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "testing" + + "github.com/mendixlabs/mxcli/sdk/microflows" + "go.mongodb.org/mongo-driver/bson" +) + +func bodyGet(doc bson.D, key string) any { + for _, e := range doc { + if e.Key == key { + return e.Value + } + } + return nil +} + +// TestWebServiceRequestBody_Arguments is the legacy half of the CE0178 fix, and +// exists to keep the two engines from drifting: the modelsdk twin +// (TestWebServiceCallAction_ArgumentsAreSimpleParameterMappings) asserts the +// same keys, values and marker. +func TestWebServiceRequestBody_Arguments(t *testing.T) { + doc := webServiceRequestBody(µflows.WebServiceCallAction{ + Arguments: []microflows.WebServiceArgument{{ + Name: "OrderId", + Path: "http%3A//www.example.com/:GetOrder|OrderId", + Expression: "$Customer/OrderId", + Checked: true, + }}, + }) + + if got := bodyGet(doc, "$Type"); got != "Microflows$SimpleRequestHandling" { + t.Fatalf("$Type = %#v", got) + } + arr, ok := bodyGet(doc, "ParameterMappings").(bson.A) + if !ok || len(arr) != 2 { + t.Fatalf("ParameterMappings = %#v, want the marker plus one mapping", bodyGet(doc, "ParameterMappings")) + } + if got, isInt := arr[0].(int32); !isInt || got != 2 { + t.Errorf("marker = %#v, want int32(2)", arr[0]) + } + pm, ok := arr[1].(bson.D) + if !ok { + t.Fatalf("mapping = %#v", arr[1]) + } + for _, want := range []struct { + key string + val any + }{ + {"$Type", "Microflows$WebServiceOperationSimpleParameterMapping"}, + {"Argument", "$Customer/OrderId"}, + {"IsChecked", true}, + {"ParameterName", ""}, + {"ParameterPath", "http%3A//www.example.com/:GetOrder|OrderId"}, + } { + if got := bodyGet(pm, want.key); got != want.val { + t.Errorf("%s = %#v, want %#v", want.key, got, want.val) + } + } +} + +// TestWebServiceRequestBody_SendMapping is the legacy half of the CE0369 fix. +// +// MappingId / MappingVariableName are the STORAGE names; modelsdk/gen binds the +// same two properties as Mapping / MappingArgumentVariableName, which mxbuild +// tolerates and Studio Pro cannot open. +func TestWebServiceRequestBody_SendMapping(t *testing.T) { + doc := webServiceRequestBody(µflows.WebServiceCallAction{ + SendMappingID: "Clients.SoapOrderExportMapping", + SendMappingVariable: "NewSaveOrder", + }) + for _, want := range []struct { + key string + val any + }{ + {"$Type", "Microflows$MappingRequestHandling"}, + {"ContentType", "Json"}, + {"MappingId", "Clients.SoapOrderExportMapping"}, + {"MappingVariableName", "NewSaveOrder"}, + } { + if got := bodyGet(doc, want.key); got != want.val { + t.Errorf("%s = %#v, want %#v", want.key, got, want.val) + } + } +} + +// TestWebServiceRequestBody_EmptyIsTheBareSimpleForm — a call with neither +// clause still writes the empty Simple body every reference call carries, so +// this change does not alter what already shipped. +func TestWebServiceRequestBody_EmptyIsTheBareSimpleForm(t *testing.T) { + doc := webServiceRequestBody(µflows.WebServiceCallAction{}) + if got := bodyGet(doc, "$Type"); got != "Microflows$SimpleRequestHandling" { + t.Fatalf("$Type = %#v", got) + } + arr, ok := bodyGet(doc, "ParameterMappings").(bson.A) + if !ok || len(arr) != 1 { + t.Fatalf("ParameterMappings = %#v, want just the marker", bodyGet(doc, "ParameterMappings")) + } +} + +// TestParseWebServiceRequestBody round-trips both variants back into the model, +// and pins that the parameter NAME comes from the path's last segment — the only +// part MDL spells. +func TestParseWebServiceRequestBody(t *testing.T) { + action := µflows.WebServiceCallAction{} + parseWebServiceRequestBody(map[string]any{ + "RequestBodyHandling": map[string]any{ + "$Type": "Microflows$SimpleRequestHandling", + "NullValueOption": "LeaveOutElement", + "ParameterMappings": []any{int32(2), map[string]any{ + "$Type": "Microflows$WebServiceOperationSimpleParameterMapping", + "Argument": "2", + "IsChecked": true, + "ParameterName": "", + "ParameterPath": "http%3A//www.example.com/:GetOrder|OrderId", + }}, + }, + }, action) + if len(action.Arguments) != 1 { + t.Fatalf("read %d arguments, want 1", len(action.Arguments)) + } + got := action.Arguments[0] + if got.Name != "OrderId" || got.Expression != "2" || !got.Checked || + got.Path != "http%3A//www.example.com/:GetOrder|OrderId" { + t.Errorf("argument = %+v", got) + } + + mapped := µflows.WebServiceCallAction{} + parseWebServiceRequestBody(map[string]any{ + "RequestBodyHandling": map[string]any{ + "$Type": "Microflows$MappingRequestHandling", + "ContentType": "Json", + "MappingId": "Clients.SoapOrderExportMapping", + "MappingVariableName": "NewSaveOrder", + }, + }, mapped) + if string(mapped.SendMappingID) != "Clients.SoapOrderExportMapping" || + mapped.SendMappingVariable != "NewSaveOrder" || + mapped.SendMappingContentType != "Json" { + t.Errorf("send mapping = %+v", mapped) + } + // The two variants differ in ARITY, so dispatching on $Type rather than on + // which fields are present is what stops one being read as the other. + if len(mapped.Arguments) != 0 { + t.Errorf("a mapping body produced %d arguments", len(mapped.Arguments)) + } +} + +// referenceSoapAction builds the fifteen-key action shape every ako/TestApp SOAP +// call carries, with mxcli's own values for the six boilerplate keys. +func referenceSoapAction() map[string]any { + return map[string]any{ + "$ID": "a", "$Type": "Microflows$CallWebServiceAction", + "ErrorHandlingType": "Rollback", + "HttpConfiguration": map[string]any{ + "$Type": "Microflows$HttpConfiguration", + "ClientCertificate": "", "CustomLocation": "", + "CustomLocationTemplate": nil, + "HttpAuthenticationPassword": "", "HttpAuthenticationUserName": "", + "HttpHeaderEntries": []any{int32(3)}, + "HttpMethod": "Post", + "OverrideLocation": false, "UseHttpAuthentication": false, + }, + "ImportedService": "Clients.OrderSoapClient", "IsValidationRequired": false, + "NewResultHandling": map[string]any{ + "$Type": "Microflows$ResultHandling", "Bind": true, + "ImportMappingCall": map[string]any{ + "$Type": "Microflows$ImportMappingCall", "Commit": "YesWithoutEvents", + "ContentType": "Xml", "ForceSingleOccurrence": false, + "ObjectHandlingBackup": "Create", "ParameterVariableName": "", + "Range": map[string]any{"$Type": "Microflows$ConstantRange", "SingleObject": true}, + "ReturnValueMapping": "Clients.SoapOrdersImportMapping", + }, + "ResultVariableName": "Orders", + "VariableType": map[string]any{"$Type": "DataTypes$ObjectType", "Entity": "Clients.Order"}, + }, + "OperationName": "GetOrder", "ProxyConfiguration": nil, + "RequestBodyHandling": map[string]any{ + "$Type": "Microflows$SimpleRequestHandling", "NullValueOption": "LeaveOutElement", + "ParameterMappings": []any{int32(2), map[string]any{ + "$Type": "Microflows$WebServiceOperationSimpleParameterMapping", + "Argument": "2", "IsChecked": true, "ParameterName": "", + "ParameterPath": "http%3A//www.example.com/:GetOrder|OrderId", + }}, + }, + "RequestHeaderHandling": map[string]any{ + "$Type": "Microflows$SimpleRequestHandling", "NullValueOption": "LeaveOutElement", + "ParameterMappings": []any{int32(2)}, + }, + "RequestProxyType": "DefaultProxy", "ServiceName": "OrdersWS", + "TimeOutExpression": "300", "UseRequestTimeOut": true, + } +} + +// TestWebServiceActionRequiresRawBSON_StructuredWhenReproducible — an action +// mxcli itself would write describes structurally rather than as base64. +// +// Before the request body was authorable this could never happen: a real call +// carries fifteen keys and only nine were admitted, so EVERY SOAP call in every +// project — Studio Pro's and mxcli's — rendered as `call web service raw '<…>'`. +func TestWebServiceActionRequiresRawBSON_StructuredWhenReproducible(t *testing.T) { + if webServiceActionRequiresRawBSON(referenceSoapAction()) { + t.Error("an action mxcli would write itself still falls back to raw") + } +} + +// TestWebServiceActionRequiresRawBSON_ValueSensitive is the regression test for +// what a describe -> exec round trip over ako/TestApp actually caught. +// +// Admitting the six boilerplate keys BY NAME would have silently normalised a +// call the moment anyone round-tripped it. Each case below is a document mxcli +// would write differently, so each must keep the byte-exact raw fallback. +func TestWebServiceActionRequiresRawBSON_ValueSensitive(t *testing.T) { + for _, tc := range []struct { + name string + mutit func(map[string]any) + }{ + // Measured: Clients.GetOrders stores SingleObject FALSE where mxcli + // writes true. No error comes of it, which is exactly why writing it + // back must not happen silently — the round trip would change the + // user's document with nothing to show for it. + {"Range.SingleObject differs", func(m map[string]any) { + rh := m["NewResultHandling"].(map[string]any) + imc := rh["ImportMappingCall"].(map[string]any) + imc["Range"] = map[string]any{"$Type": "Microflows$ConstantRange", "SingleObject": false} + }}, + // Measured: Clients.SaveOrder binds $IsSaved with NO import mapping and + // a DataTypes$BooleanType — the OPERATION's return type, which lives in + // the WSDL. Written back as VoidType it is CE0366 + CE6011. + {"result type comes from the WSDL, not a mapping", func(m map[string]any) { + m["NewResultHandling"] = map[string]any{ + "$Type": "Microflows$ResultHandling", "Bind": true, + "ImportMappingCall": nil, + "ResultVariableName": "IsSaved", + "VariableType": map[string]any{"$Type": "DataTypes$BooleanType"}, + } + }}, + {"HTTP authentication configured", func(m map[string]any) { + m["HttpConfiguration"].(map[string]any)["UseHttpAuthentication"] = true + }}, + {"custom location", func(m map[string]any) { + m["HttpConfiguration"].(map[string]any)["CustomLocation"] = "https://elsewhere/" + }}, + {"a SOAP header is configured", func(m map[string]any) { + m["RequestHeaderHandling"].(map[string]any)["ParameterMappings"] = []any{int32(2), + map[string]any{"$Type": "Microflows$WebServiceOperationSimpleParameterMapping"}} + }}, + {"validation required", func(m map[string]any) { m["IsValidationRequired"] = true }}, + {"non-default proxy", func(m map[string]any) { m["RequestProxyType"] = "NoProxy" }}, + {"timeout disabled", func(m map[string]any) { m["UseRequestTimeOut"] = false }}, + // A per-parameter export mapping has no MDL spelling at all. + {"advanced parameter mapping", func(m map[string]any) { + m["RequestBodyHandling"].(map[string]any)["ParameterMappings"] = []any{int32(2), + map[string]any{"$Type": "Microflows$WebServiceOperationAdvancedParameterMapping"}} + }}, + // Without a "|" the parameter name MDL spells cannot be recovered, so + // the write path could not rebuild the same path. + {"parameter path with no name segment", func(m map[string]any) { + pms := m["RequestBodyHandling"].(map[string]any)["ParameterMappings"].([]any) + pms[1].(map[string]any)["ParameterPath"] = "http%3A//www.example.com/:GetOrder" + }}, + {"unknown key entirely", func(m map[string]any) { m["SomethingNew"] = 1 }}, + } { + t.Run(tc.name, func(t *testing.T) { + m := referenceSoapAction() + tc.mutit(m) + if !webServiceActionRequiresRawBSON(m) { + t.Error("describes structurally, so a round trip would silently rewrite it") + } + }) + } +} From aaaf5b23bbd4a03150e483af0ba7988c3dacb8bb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 20:40:18 +0000 Subject: [PATCH 03/18] fix(alter-page): resolve every data source kind in one walk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ALTER PAGE REPLACE/INSERT built the new widget's attribute bindings in the wrong scope in two shapes, both silent through `mxcli check --references` and `exec`: - inside `dataview dv (datasource: selection lv)` the binding re-scoped to the OUTER data view's entity — CE1613 "the selected attribute no longer exists", and `describe page` masks it by printing the bare attribute name; - inside a Gallery/DataGrid 2 sourced by a microflow or nanoflow the binding was dropped — CE0402 "No value specified", `describe` shows `ContentParams: [{1} = ]`. The mutator resolved a widget's scope in two separate walks that each knew a different subset of Mendix's ten Forms$*Source kinds — the third time that split has written a wrong binding (FINDINGS #55, #935). Forms$ListenTargetSource carries no EntityRef at all, only the listen target's NAME, so the entity walk left the context at the enclosing data view; and the flow walk read only a widget's top-level DataSource key, so a pluggable list — whose source sits in Object.Properties[datasource] — was invisible to it, as was an object-list item's own widgets. One resolver over one walk now. The ten kinds divide exactly three ways (seven carry an EntityRef, two are flows, one borrows the scope of the widget it listens to), so resolveSourceScope can be complete where a per-kind patch never is. A nearer source that resolves to no entity SHADOWS the enclosing one instead of letting it be inherited — that inheritance is what produced a plausible wrong entity, and it also mis-scoped a flow-sourced list nested in an entity-bound data view, a case the report did not name. Measured on mxbuild 11.10.0 with the new repro: 2 × CE1613 and three `` bindings before, 0 errors and every binding entity-qualified after; each half proven load-bearing by stubbing it alone and rebuilding the CLI. Note the trap that cost a wrong first reading — a CE1613 suppresses the CE0402s in the same `mx check` run. The issue's own second repro (a Gallery over a database source) no longer reproduced; #935 had fixed that one. Fixes mendixlabs/mxcli#1076 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01L45JFiJ6y58ftg2zWVEq7h --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .claude/skills/mendix/alter-page/SKILL.md | 10 + CHANGELOG.md | 10 + docs-wiki/bug-patterns/mutator-addressing.md | 21 ++ ...76-alter-page-selection-and-flow-scope.mdl | 155 ++++++++ mdl/backend/pagemutator/mutator.go | 344 ++++++++---------- .../mutator_selection_source_test.go | 263 +++++++++++++ 7 files changed, 620 insertions(+), 184 deletions(-) create mode 100644 mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl create mode 100644 mdl/backend/pagemutator/mutator_selection_source_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index fee1d0182..c68238dae 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -89,3 +89,4 @@ {"area": "mdl/backend", "date": "2026-09-09", "symptom": "`call web service` (legacy SOAP) on the DEFAULT engine: `mxcli exec 06b-soap-examples.mdl` reported success on all three microflows and `mx check` (11.13.0) then failed the project with `[CE0008] \"No action defined.\" at Action activity 'Activity'` plus two `[CE0109] \"Undefined variable 'Root'.\"`. Only reachable by rerunning with MXCLI_ENGINE=legacy, which is why legacy was still the documented fallback", "cause": "The codec engine READ the action (`actionFromGen` \u2192 `*microflows.WebServiceCallAction`, with a raw fallback) but `microflowActionToGen` had no case for it, so it hit `default: return nil` and the enclosing ActionActivity serialized with no action at all. The #850 shape: a missing WRITE case is not an unsupported feature, it is a silent drop that exec reports as success. The CE0109s are knock-on \u2014 the dropped action never bound $Root", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go` (new), `microflow_write.go` (switch case)", "insight": "**Mirror the legacy serializer, and prove it by diffing the two engines' documents \u2014 do not re-derive the shape from the metamodel.** There is no Studio Pro-authored SOAP document in this repo, so legacy's output is the only reference that exists and is also what users' projects already contain. Method: exec the same script on each engine, `mxcli bson dump` both, normalise the random $IDs, diff. Three discrepancies fell out that no amount of reading would have shown, all in how the codec emits NULLS and MARKERS: (1) a Part property with no child encodes to nil and the encoder then SKIPS the key (`if val != nil`), so an unset part is an ABSENT key, never a null one \u2014 carry the null as a primitive `bson.Null{}` value instead, which marshals in place; (2) `codec.TypeDefaults.NullFields` does emit the key but APPENDS it after every property, so it cannot reproduce alphabetical key order; (3) both NullFields and list markers are registered per `$Type` and several types are SHARED between writers \u2014 `Microflows$HttpConfiguration` needs HttpHeaderEntries marker 3 for SOAP and 2 for REST, and legacy writes CustomLocationTemplate as null for SOAP but omits it for REST, so a global registration would have silently changed the REST path. Write those explicitly per call site. (The package already carries one such collision: `Microflows$HttpHeaderEntry` is registered 2 in microflow_write.go and 3 in odata_write.go, decided by file order.) Verified: the two engines' CallWebServiceAction documents are now identical key-for-key and value-for-value; `mx check` goes 3 structural errors \u2192 0, leaving only the script's 4 deliberate CE1613 dangling refs, which is exactly legacy's output. Revert control: removing the switch case fails all six unit tests with the CE0008 message. The doctype engineScriptSkip for modelsdk/06b was removed and the script now runs on BOTH engines", "refs": []} {"area": "mdl-backend", "date": "2026-09-10", "symptom": "`mxcli diff-local` on an MPR v2 project fails with `Error: mprcontents directory not found` while mprcontents/ exists and is populated; `MXCLI_ENGINE=legacy` works", "cause": "The modelsdk engine (the default) never overrode `Backend.ContentsDir()`, so it fell through to the generated `unimplemented` stub and returned \"\". diff-local reads \"\" as 'not a v2 project'.", "file": "mdl/backend/modelsdk/backend.go", "insight": "gen_unimplemented.go's promise that an unoverridden method 'fails loudly rather than silently dropping data' is CONDITIONAL on the method having an error to fail through: the generated body is `errUnimplemented` only when a result is `error`, a panic when there are no results at all, and a silent `var r0 T; return r0` otherwise. ContentsDir is in the third bucket and its zero value is a MEANINGFUL in-band answer (\"\" == MPR v1), so the missing implementation was indistinguishable from a v1 project rather than looking like a bug. The detectable signature was the contradiction between two questions the same command asks: Version() (implemented) says 2, ContentsDir() says v1. Guard added in mdl/backend/modelsdk/unimplemented_silent_test.go \u2014 reflect over FullBackend for error-less methods, go/parser the package for methods actually declared on *Backend, since reflection cannot tell a promoted method from an override (Go synthesises a wrapper named (*Backend).X for both). It immediately found a second one, InvalidateCache (a latent panic, no caller today).", "refs": ["mendixlabs/mxcli#1080"]} {"area": "mdl/backend", "date": "2026-09-10", "symptom": "SOAP `call web service` writes a document mxbuild accepts and Studio Pro would not have written. A SEND MAPPING is silently DROPPED by both engines \u2014 `send mapping Mod.Export` parses, `mxcli check` passes, `exec` reports success, and nothing in the stored action references the mapping. Operation ARGUMENTS are dropped the same way", "cause": "sdk/mpr.serializeWebServiceCallAction was written without a Studio Pro reference and hardcodes five things it cannot know, and the codec engine's new writer reproduced it deliberately for parity. Measured against three Studio Pro-authored calls in ako/TestApp (Mendix 11.14.0, Clients.GetOrders / GetCustomerOrders / SaveOrder): ServiceName is the WSDL SERVICE name (\"OrdersWS\") not the local part of the imported service's qualified name (\"OrderSoapClient\"); ImportMappingCall.ContentType is \"Xml\" for a SOAP import mapping, not \"Json\"; Range.SingleObject follows cardinality (false for a list) rather than being always true; VariableType is the real result type (DataTypes$ObjectType with an Entity, DataTypes$BooleanType) rather than always DataTypes$VoidType; and a send mapping is Microflows$MappingRequestHandling {ContentType, MappingId, MappingVariableName}. Arguments live in RequestBodyHandling.ParameterMappings as Microflows$WebServiceOperationSimpleParameterMapping entries keyed by an escaped ParameterPath (\"http%3A//www.example.com/:GetOrder|OrderId\")", "file": "`sdk/mpr/writer_microflow_actions.go` (serializeWebServiceCallAction), `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**A guessed type name in a comment becomes a permanent refusal.** Legacy refused send mappings citing `Mendix$AdvancedRequestHandling`, said it 'requires a Studio Pro-generated example to determine the correct type storage name', and that refusal then shipped for as long as nobody went looking. The real type is `Microflows$MappingRequestHandling` \u2014 which THIS CODEBASE ALREADY WRITES for REST result/request handling \u2014 and the guessed name occurs in none of the three reference documents. The lesson is not about SOAP: when a writer refuses because a storage name is unknown, check whether a sibling feature already writes it before treating the refusal as a standing constraint. **Second, and the reason this was found at all: 'no reference exists' is a claim about where you looked.** The parity work asserted that no Studio Pro-authored SOAP document existed to pin against and used that to justify mirroring legacy; one existed in a separate repo the whole time (ako/TestApp, which carries both a consumed client and a published service). Byte-parity with what ships is a legitimate goal for a change scoped to stopping a silent drop \u2014 it is NOT evidence the shape is right, and conflating the two is how six defects got a passing test. Where a reference project exists, name it in the code so the next reader does not repeat the search", "refs": []} +{"area": "mdl/backend", "date": "2026-09-11", "symptom": "`ALTER PAGE REPLACE`/`INSERT` inside a data view bound `datasource: selection ` re-scopes the new widget's attribute binding to the OUTER data view's entity (**CE1613** \"The selected attribute 'Mod.Outer.Attr' no longer exists\"), and inside a Gallery/DataGrid 2 sourced by a **microflow/nanoflow** drops it entirely (**CE0402** \"No value specified\", `describe` shows `ContentParams: [{1} = ]`). `mxcli check --references` and `exec` both report success; `CREATE PAGE` binds the same widget in the same position correctly", "cause": "The mutator resolved a widget's scope in TWO walks that each knew a different subset of the ten Forms$*Source kinds. `Forms$ListenTargetSource` carries no EntityRef at all \u2014 only the listen target's NAME \u2014 so the entity walk saw no source on the selection data view and left the context at the enclosing one. The flow walk (`findNearestDataSourceDoc`) read only a widget's TOP-LEVEL `DataSource` key, so a pluggable list \u2014 whose source sits at `Object.Properties[datasource].Value.DataSource` \u2014 contributed nothing, and its `Objects[].Properties[].Value.Widgets` descent (the one the entity walk gained in #935) was missing too", "file": "`mdl/backend/pagemutator/mutator.go` (`resolveSourceScope`/`resolveSourceScopeVia`, `listenTargetDataSource`, `widgetOwnDataSourceDoc`, `pluggableDataSourceDoc`; `EnclosingEntity`/`EnclosingEntityForChildren`/`EnclosingDataSourceFlow` now share the one walk `findNearestDataSourceDoc`, and `findEnclosingEntityContext` + its two helpers are deleted)", "insight": "**Count the source kinds before fixing one.** `generated/metamodel/types.go`'s `DataSource is implemented by` list closes the set at ten, and they divide exactly three ways \u2014 seven carry an EntityRef, two are flows, one (ListenTarget) borrows the scope of the widget it names \u2014 so one resolver can be complete, where three successive per-kind patches (FINDINGS #55 association+flow, #935 pluggable, this one) each left a hole. **A nearer source that resolves to no entity must SHADOW the outer one**: inheriting is what wrote the wrong entity, and it also mis-scoped a flow-sourced list nested in an entity-bound data view \u2014 a case the report did not name and the old code got wrong. The listen target is found by a shape-independent search for \"a document with this Name that has a data source\", which is what makes it work when the target is a pluggable widget keeping its source three levels inside its Object; a visited-set guards a hand-written listen cycle. **Measurement trap: a CE1613 SUPPRESSES the CE0402s in the same `mx check` run** \u2014 the first reading said mxbuild tolerated the unbound parameter, and the CE0402s only appeared once the re-scoped binding was fixed, so count bindings in `describe`, not errors. The issue's own second repro (a Gallery over a DATABASE source) no longer reproduced \u2014 #935 had fixed it \u2014 and the live defect was its flow-sourced variant, so re-measure a report against main before trusting its class. Tests `mdl/backend/pagemutator/mutator_selection_source_test.go` (7, incl. dangling/cyclic listen targets and the shadowing control); repro `mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl` \u2014 2 \u00d7 CE1613 + 3 unbound before, 0 errors after, on mxbuild 11.10.0. Each half proven load-bearing by stubbing it alone and rebuilding the CLI", "refs": ["mendixlabs/mxcli#1076", "#55", "#935"], "ce": ["CE0402", "CE1613"]} diff --git a/.claude/skills/mendix/alter-page/SKILL.md b/.claude/skills/mendix/alter-page/SKILL.md index bb6f783cf..ca76c6cdb 100644 --- a/.claude/skills/mendix/alter-page/SKILL.md +++ b/.claude/skills/mendix/alter-page/SKILL.md @@ -239,6 +239,16 @@ dataview's entity as their context. Supported on simple containers (container, dataview, groupbox, scroll-container region); for a layout grid or tab container, insert relative to a widget inside the target column/tab instead. +**The context comes from the nearest enclosing data source, whatever kind it is** +— a database or association source, a microflow/nanoflow source (the entity is +the flow's return type), or `datasource: selection `, which takes the +entity of the list it listens to. A bare attribute in the inserted or replaced +widget resolves against that entity, exactly as it would in `create page`. When +no enclosing source can be resolved, the binding is written unset rather than +guessed at — `describe page` then prints ``, and mxbuild reports +`CE0402 "No value specified."`, so re-describe the page after an ALTER that +moves data-bound widgets. + ### DROP - Remove Widgets ```sql diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d367eae7..cbcf6464c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`ALTER PAGE` bound the replacement widget against the wrong data context** (mendixlabs/mxcli#1076) — `replace txtPreviewPeriod with { … }` inside a data view bound `datasource: selection lvVersions` re-scoped the binding to the **outer** data view's entity (`[CE1613] "The selected attribute 'Bug.Dashboard.PeriodLabel' no longer exists."`), and the same statement inside a Gallery or DataGrid 2 sourced by a **microflow/nanoflow** dropped the binding entirely (`[CE0402] "No value specified."`, `describe` renders `ContentParams: [{1} = ]`). `mxcli check --references` and `exec` both reported success; `CREATE PAGE` binds the same widget in the same position correctly, so the defect was in the ALTER walk alone. + + A widget's scope was resolved by **two separate walks that each knew a different subset** of Mendix's ten `Forms$*Source` kinds, which is the third time that split has produced a wrong binding (association and flow sources in FINDINGS #55, pluggable widgets in ako/mxcli#935). `Forms$ListenTargetSource` carries no `EntityRef` at all — only the listen target's *name* — so the entity walk saw no source and left the context at the enclosing data view; and the flow walk read only a widget's **top-level** `DataSource` key, so a pluggable list, whose source sits in `Object.Properties[datasource]`, was invisible to it. + + There is now one resolver over one walk. The ten kinds divide exactly three ways — seven carry an `EntityRef`, two are flows, one is a selection — so `resolveSourceScope` reads the entity, the flow's qualified name, or the listen target's own source (following it into a pluggable widget's `Object`, with a cycle guard), from wherever that widget kind stores it. A nearer source that resolves to no entity now **shadows** the enclosing one rather than letting it be inherited: inheriting is precisely how the wrong entity got written, and it also mis-scoped a flow-sourced list nested inside an entity-bound data view — a case the report did not name. + + The issue's second repro — a Gallery over a **database** source — no longer reproduces; ako/mxcli#935 fixed that one. What remains, and is fixed here, is the flow-sourced variant of it, plus a selection data view listening to a pluggable list. + + Measured on mxbuild 11.10.0 against a project sitting at 0 errors, with `mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl`: 2 × CE1613 and three `` bindings before, 0 errors and every binding entity-qualified after. Note the measurement trap, which cost a wrong reading first time round — **a CE1613 suppresses the CE0402s in the same `mx check` run**, so the unbound widgets look clean until the re-scoped one is fixed. + - **An `ALTER WORKFLOW` insert aimed at the wrong activity kind wrote a model Mendix cannot load** (ako/mxcli#415) — `alter workflow M.W insert outcome 'X' on decision9 { };` printed `Altered workflow`, and the project then failed to **load**: `mx check` died at *"Loading the mpr file"* with `System.InvalidCastException: Unable to cast object of type 'UserTaskOutcome' to type 'ConditionOutcome'`, before validating anything, so Studio Pro would not open it either. The MDL-WF04 class — the blast radius is the whole project, not one document — and every mxcli-side command reported success, which is what let it survive any number of later scripts. An activity's outcome list is **typed**, and each inserting op writes exactly one outcome type into it: `INSERT OUTCOME` a `UserTaskOutcome`, `INSERT PATH` a `ParallelSplitOutcome`, `INSERT CONDITION` a `…ConditionOutcome`. `generated/metamodel` types the receiving list per activity, and none of the three ops checked what it was pointed at. diff --git a/docs-wiki/bug-patterns/mutator-addressing.md b/docs-wiki/bug-patterns/mutator-addressing.md index 543487543..74ebc962e 100644 --- a/docs-wiki/bug-patterns/mutator-addressing.md +++ b/docs-wiki/bug-patterns/mutator-addressing.md @@ -94,6 +94,27 @@ bug: it turns a silent no-op into a project Studio Pro cannot open. "widget not found" turned out to be two independent defects, and the fix for the reported nesting alone would not have made the reporter's command work. +**Naming the node is half the problem; the other half is naming its scope.** A +widget inserted or replaced in place has to be built in the data context its new +position implies, and the mutator reads that context out of raw BSON rather than +from the model the CREATE builder carries. Three separate reports — an +association-bound list, a widget inside a pluggable column, a selection-driven +data view — were all the same defect: the walk understood some of Mendix's data +source kinds and not others, so the widget was built against the *enclosing* +entity, or against none. The symptom is the worst shape in this family, because +`mxcli check`, `exec` and `describe` all report normally and the model fails at +`mx check` or simply renders nothing. + +Two things make that class closable rather than recurring. The **set of source +kinds is closed and small** — `generated/metamodel`'s `DataSource is implemented +by` list has ten, dividing into "carries an entity", "is a flow whose return type +is the entity", and "borrows another widget's scope" — so one resolver can be +demonstrably complete where a per-kind patch never is. And **a source that +resolves to nothing must shadow the enclosing entity, not fall through to it**: +falling through is what silently produced a *plausible* wrong entity instead of +an obvious empty one, which is why the failure reached mxbuild rather than the +author. + ## See also - [fix-issue findings](../../.claude/skills/fix-issue/findings/) — the individual diff --git a/mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl b/mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl new file mode 100644 index 000000000..731507270 --- /dev/null +++ b/mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl @@ -0,0 +1,155 @@ +-- ============================================================================ +-- mendixlabs/mxcli#1076: ALTER PAGE REPLACE / INSERT resolves the new widget's +-- attribute bindings against the wrong data context +-- ============================================================================ +-- +-- Symptom: `mxcli check --references` and `mxcli exec` both report success, and +-- the model then fails at `mx check` (or renders nothing) in two shapes: +-- +-- 1. a data view bound `datasource: selection ` — the binding silently +-- re-scoped to the OUTER data view's entity: +-- [CE1613] "The selected attribute 'BugTest1076.Dashboard.PeriodLabel' +-- no longer exists." at Text 'txtPreviewPeriod' +-- DESCRIBE masks this one: it prints the short attribute name, which reads +-- exactly as it did before. +-- 2. a PLUGGABLE list (Gallery, DataGrid 2) sourced by a microflow/nanoflow — +-- the binding was dropped entirely: +-- [CE0402] "No value specified." at Text 'txtName2' +-- DESCRIBE shows `ContentParams: [{1} = ]`. +-- +-- Root cause: the mutator resolved a widget's scope in two separate walks that +-- each knew a different subset of the ten Forms$*Source kinds. +-- Forms$ListenTargetSource carries no EntityRef at all (only the listen +-- target's NAME), so the entity walk left the context at the outer data view; +-- and the flow walk read only a widget's TOP-LEVEL DataSource key, so a +-- pluggable list — whose source sits in Object.Properties[datasource] — was +-- invisible to it. +-- +-- Fix: one resolver (resolveSourceScope in mdl/backend/pagemutator/mutator.go) +-- over one walk. It reads the entity, the flow, or the listen target's own +-- source, from wherever this widget kind keeps it; a source that resolves to no +-- entity now SHADOWS the enclosing one instead of letting it be inherited. +-- +-- Verify (measured on mxbuild 11.10.0): exec this script, then `mx check` → +-- 0 errors, and every `describe page` below binds each ContentParams to its own +-- entity. Against the pre-fix binary the same script gives 2 × CE1613 and three +-- `` bindings. +-- +-- Measurement trap: a CE1613 elsewhere in the project SUPPRESSES the CE0402s of +-- the unbound ones in the same `mx check` run — it reports the two CE1613s and +-- nothing else, and the three CE0402s appear only once those are fixed. Count +-- the bindings in `describe`, not the errors. +-- ============================================================================ + +create module BugTest1076; +/ + +create persistent entity BugTest1076.Dashboard ( Name: String ); +create persistent entity BugTest1076.DashboardVersion ( PeriodLabel: String ); +create association BugTest1076.DashboardVersion_Dashboard + from BugTest1076.DashboardVersion to BugTest1076.Dashboard; +create persistent entity BugTest1076.Item ( Name: String ); +/ + +create microflow BugTest1076.DS_Items () returns list of BugTest1076.Item as $out +begin + retrieve $out from BugTest1076.Item; + return $out; +end; +/ + +-- --------------------------------------------------------------------------- +-- 1. Selection-driven data view nested inside a page-level data view. +-- dvSelected's children are DashboardVersion, never Dashboard. +-- --------------------------------------------------------------------------- +create or replace page BugTest1076.Dashboard_View +( Title: 'Dashboard', Layout: Atlas_Core.Atlas_Default, + Params: { $Dashboard: BugTest1076.Dashboard } ) +{ + dataview dvDashboard (datasource: $Dashboard) { + listview lvVersions (datasource: association BugTest1076.DashboardVersion_Dashboard) { + dynamictext txtRow (content: '{1}', ContentParams: [{1} = PeriodLabel]) + } + dataview dvSelected (datasource: selection lvVersions) { + dynamictext txtPreviewPeriod (content: 'Period: {1}', ContentParams: [{1} = PeriodLabel]) + } + } +} +/ + +alter page BugTest1076.Dashboard_View { + replace txtPreviewPeriod with { + dynamictext txtPreviewPeriod (content: 'Preview {1}', ContentParams: [{1} = PeriodLabel]) + } +} +/ + +-- INSERT INTO the selection data view takes the same scope as REPLACE. +alter page BugTest1076.Dashboard_View { + insert into dvSelected { + dynamictext txtPreviewProbe (content: 'P {1}', ContentParams: [{1} = PeriodLabel]) + } +} +/ + +-- --------------------------------------------------------------------------- +-- 2. Pluggable list sourced by a microflow: its entity is the flow's RETURN +-- type, and its source lives inside the widget's Object, not beside it. +-- --------------------------------------------------------------------------- +create or replace page BugTest1076.Page_MfGallery +( Title: 'MfGallery', Layout: Atlas_Core.Atlas_Default ) +{ + gallery galMf (datasource: microflow BugTest1076.DS_Items) { + template t1 { + container ctnCard { + dynamictext txtName (content: '{1}', ContentParams: [{1} = Name]) + } + } + } +} +/ + +alter page BugTest1076.Page_MfGallery { + replace txtName with { + container ctnWrap { + dynamictext txtName2 (content: '{1}', ContentParams: [{1} = Name]) + } + } +} +/ + +alter page BugTest1076.Page_MfGallery { + insert after txtName2 { + dynamictext txtNameProbe (content: 'Q {1}', ContentParams: [{1} = Name]) + } +} +/ + +-- --------------------------------------------------------------------------- +-- 3. Both halves at once: a selection data view listening to a PLUGGABLE list. +-- Resolving it means following the listen target INTO its Object. +-- --------------------------------------------------------------------------- +create or replace page BugTest1076.Page_SelGallery +( Title: 'SelGallery', Layout: Atlas_Core.Atlas_Default ) +{ + gallery galItems (datasource: database BugTest1076.Item) { + template t1 { + dynamictext txtCard (content: '{1}', ContentParams: [{1} = Name]) + } + } + dataview dvSel (datasource: selection galItems) { + dynamictext txtSel (content: 'S {1}', ContentParams: [{1} = Name]) + } +} +/ + +alter page BugTest1076.Page_SelGallery { + replace txtSel with { + dynamictext txtSel (content: 'S2 {1}', ContentParams: [{1} = Name]) + } +} +/ + +describe page BugTest1076.Dashboard_View; +describe page BugTest1076.Page_MfGallery; +describe page BugTest1076.Page_SelGallery; diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index 7c61e94be..7205b9b1a 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -949,8 +949,12 @@ func (m *Mutator) SetPluggableProperty(widgetRef string, propKey string, opName return fmt.Errorf("pluggable property %q not found on widget %q", propKey, widgetRef) } +// EnclosingEntity returns the entity context that applies to a widget's +// SIBLINGS — what REPLACE and INSERT BEFORE/AFTER build against — i.e. the +// entity of the nearest enclosing data source. func (m *Mutator) EnclosingEntity(widgetRef string) string { - return findEnclosingEntityContext(m.rawData, widgetRef) + ds, _ := findNearestDataSourceDoc(m.rawData, widgetRef) + return resolveSourceScope(m.rawData, ds).Entity } // EnclosingDataSourceFlow returns the microflow/nanoflow qualified name of the @@ -967,8 +971,9 @@ func (m *Mutator) EnclosingEntity(widgetRef string) string { func (m *Mutator) EnclosingDataSourceFlow(widgetRef string, forChildren bool) (microflow, nanoflow string) { if forChildren { if result := m.widgetFinder(m.rawData, widgetRef); result != nil { - if ds := bsonnav.DGetDoc(result.widget, "DataSource"); ds != nil { - return flowFromDataSourceDoc(ds) + if ds := widgetOwnDataSourceDoc(result.widget); ds != nil { + scope := resolveSourceScope(m.rawData, ds) + return scope.Microflow, scope.Nanoflow } } } @@ -976,7 +981,8 @@ func (m *Mutator) EnclosingDataSourceFlow(widgetRef string, forChildren bool) (m if !ok { return "", "" } - return flowFromDataSourceDoc(ds) + scope := resolveSourceScope(m.rawData, ds) + return scope.Microflow, scope.Nanoflow } // flowFromDataSourceDoc extracts the microflow/nanoflow qualified name from a @@ -1004,39 +1010,135 @@ func (m *Mutator) EnclosingEntityForChildren(widgetRef string) string { if result == nil { return "" } - if ent := extractEntityFromDataSource(result.widget); ent != "" { - return ent + // A widget that declares a source of its own governs its children, even when + // that source names no entity: a flow-sourced or selection-bound list whose + // scope cannot be read here must SHADOW the enclosing entity rather than let + // it be inherited — inheriting it is how a binding silently re-scoped to the + // outer data view (#1076). + if ds := widgetOwnDataSourceDoc(result.widget); ds != nil { + return resolveSourceScope(m.rawData, ds).Entity } - if ent := extractPluggableDataSourceEntity(result.widget); ent != "" { - return ent + return m.EnclosingEntity(widgetRef) +} + +// --------------------------------------------------------------------------- +// Data source resolution +// --------------------------------------------------------------------------- + +// sourceScope is what one data source hands to the widgets under it: the entity +// it names, or the qualified name of the microflow/nanoflow whose RETURN type is +// that entity — which lives in the flow document, so only a caller holding the +// model can finish that half. +// +// Mendix has ten data source kinds and they divide exactly three ways: seven +// carry an EntityRef (Association, CustomWidgetXPath, DataView, GridXPath, +// ImageViewer, ListViewXPath, ReferenceSet — see the `DataSource is implemented +// by` list in generated/metamodel/types.go), two are flows (Microflow, +// Nanoflow), and one — ListenTargetSource — carries neither and borrows the +// scope of the widget it listens to. Every kind is therefore resolved here, and +// a source that still resolves to nothing means the model names nothing, not +// that this walk has another shape left to learn. Each time one kind was handled +// somewhere and not elsewhere, the result was a binding written against the +// wrong entity: association and flow sources (FINDINGS #55), pluggable widgets +// (#935), selection sources and pluggable flow sources (#1076). +type sourceScope struct { + Entity string + Microflow string + Nanoflow string +} + +// resolveSourceScope reads one widget's "DataSource" document. root is the whole +// unit, needed only to follow a selection source to its listen target. +func resolveSourceScope(root bson.D, ds bson.D) sourceScope { + return resolveSourceScopeVia(root, ds, nil) +} + +// resolveSourceScopeVia carries the listen targets already followed, so a +// selection chain that loops back on itself terminates instead of recursing +// forever. Studio Pro will not author that, but a hand-written ALTER can. +func resolveSourceScopeVia(root bson.D, ds bson.D, seen map[string]bool) sourceScope { + if ds == nil { + return sourceScope{} + } + if entity := entityFromEntityRef(bsonnav.DGetDoc(ds, "EntityRef")); entity != "" { + return sourceScope{Entity: entity} + } + if mf, nf := flowFromDataSourceDoc(ds); mf != "" || nf != "" { + return sourceScope{Microflow: mf, Nanoflow: nf} + } + // Forms$ListenTargetSource — `dataview dv (datasource: selection lv)`. It + // stores the target widget's NAME and nothing else, so its scope is whatever + // the target's own source resolves to, which may in turn be an association, + // a flow or another selection. + target := bsonnav.DGetString(ds, "ListenTarget") + if target == "" || seen[target] { + return sourceScope{} + } + if seen == nil { + seen = make(map[string]bool, 2) + } + seen[target] = true + return resolveSourceScopeVia(root, listenTargetDataSource(root, target), seen) +} + +// listenTargetDataSource returns the data source of the widget a selection +// source names. A listen target is addressed by name and need not be a sibling +// of the listening widget, so it is searched for over the whole unit; keying the +// search on "carries this Name and a data source" rather than on a list of +// container shapes is what keeps it working for a pluggable list, whose source +// sits three levels inside its Object. +func listenTargetDataSource(root bson.D, name string) bson.D { + var found bson.D + var walk func(v any) + walk = func(v any) { + if found != nil { + return + } + switch node := v.(type) { + case bson.D: + if bsonnav.DGetString(node, "Name") == name { + if ds := widgetOwnDataSourceDoc(node); ds != nil { + found = ds + return + } + } + for _, kv := range node { + walk(kv.Value) + } + case bson.A: + for _, elem := range node { + walk(elem) + } + } } - return findEnclosingEntityContext(m.rawData, widgetRef) + walk(root) + return found } -// widgetOwnEntity returns the entity a widget contributes to its descendants, -// whatever kind of widget it is. A plain Forms$ container keeps its source at -// the top level; a pluggable one (DataGrid2, Gallery) keeps it in -// Object.Properties under the schema's "datasource" key. EnclosingEntityForChildren -// already consulted both, but the recursive walk consulted only the first, so a -// widget nested under a pluggable list inherited the PAGE's context instead of -// the list's. -func widgetOwnEntity(wDoc bson.D) string { - if ent := extractEntityFromDataSource(wDoc); ent != "" { - return ent +// widgetOwnDataSourceDoc returns the widget's own "DataSource" document, +// wherever its kind keeps it: at the top level for a plain Forms$ widget, and +// under Object.Properties[datasource].Value for a pluggable one (Gallery, +// DataGrid 2). Reading only the first is what made a flow-sourced gallery +// contribute nothing, so a widget replaced inside its template was written with +// no binding at all (#1076). +func widgetOwnDataSourceDoc(wDoc bson.D) bson.D { + if ds := bsonnav.DGetDoc(wDoc, "DataSource"); ds != nil { + return ds } - return extractPluggableDataSourceEntity(wDoc) + return pluggableDataSourceDoc(wDoc) } -// extractPluggableDataSourceEntity walks a CustomWidget's Object.Properties[] -// looking for a "datasource" property and returns the EntityRef.Entity if any. -func extractPluggableDataSourceEntity(widgetDoc bson.D) string { +// pluggableDataSourceDoc walks a CustomWidget's Object.Properties[] for the +// property the widget's schema keys "datasource", and returns its DataSource +// document. +func pluggableDataSourceDoc(widgetDoc bson.D) bson.D { obj := bsonnav.DGetDoc(widgetDoc, "Object") if obj == nil { - return "" + return nil } propKeyMap := buildPropKeyMap(widgetDoc) if len(propKeyMap) == 0 { - return "" + return nil } for _, prop := range bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) { propDoc, ok := prop.(bson.D) @@ -1051,15 +1153,11 @@ func extractPluggableDataSourceEntity(widgetDoc bson.D) string { if valDoc == nil { continue } - dsDoc := bsonnav.DGetDoc(valDoc, "DataSource") - if dsDoc == nil { - continue - } - if entity := entityFromEntityRef(bsonnav.DGetDoc(dsDoc, "EntityRef")); entity != "" { - return entity + if dsDoc := bsonnav.DGetDoc(valDoc, "DataSource"); dsDoc != nil { + return dsDoc } } - return "" + return nil } func (m *Mutator) WidgetScope() map[string]model.ID { @@ -1687,150 +1785,6 @@ func sanitizeColumnName(caption string) string { // Entity context extraction // --------------------------------------------------------------------------- -// findEnclosingEntityContext walks the raw BSON tree to find the entity context. -func findEnclosingEntityContext(rawData bson.D, widgetName string) string { - if formCall := bsonnav.DGetDoc(rawData, "FormCall"); formCall != nil { - args := bsonnav.DGetArrayElements(bsonnav.DGet(formCall, "Arguments")) - for _, arg := range args { - argDoc, ok := arg.(bson.D) - if !ok { - continue - } - if ctx := findEntityContextInWidgets(argDoc, "Widgets", widgetName, ""); ctx != "" { - return ctx - } - } - } - if ctx := findEntityContextInWidgets(rawData, "Widgets", widgetName, ""); ctx != "" { - return ctx - } - if widgetContainer := bsonnav.DGetDoc(rawData, "Widget"); widgetContainer != nil { - if ctx := findEntityContextInWidgets(widgetContainer, "Widgets", widgetName, ""); ctx != "" { - return ctx - } - } - return "" -} - -func findEntityContextInWidgets(parentDoc bson.D, key string, widgetName string, currentEntity string) string { - elements := bsonnav.DGetArrayElements(bsonnav.DGet(parentDoc, key)) - for _, elem := range elements { - wDoc, ok := elem.(bson.D) - if !ok { - continue - } - if bsonnav.DGetString(wDoc, "Name") == widgetName { - return currentEntity - } - entityCtx := currentEntity - if ent := widgetOwnEntity(wDoc); ent != "" { - entityCtx = ent - } - if ctx := findEntityContextInChildren(wDoc, widgetName, entityCtx); ctx != "" { - return ctx - } - } - return "" -} - -func findEntityContextInChildren(wDoc bson.D, widgetName string, currentEntity string) string { - typeName := bsonnav.DGetString(wDoc, "$Type") - - if ctx := findEntityContextInWidgets(wDoc, "Widgets", widgetName, currentEntity); ctx != "" { - return ctx - } - if ctx := findEntityContextInWidgets(wDoc, "FooterWidgets", widgetName, currentEntity); ctx != "" { - return ctx - } - if strings.Contains(typeName, "LayoutGrid") { - rows := bsonnav.DGetArrayElements(bsonnav.DGet(wDoc, "Rows")) - for _, row := range rows { - rowDoc, ok := row.(bson.D) - if !ok { - continue - } - cols := bsonnav.DGetArrayElements(bsonnav.DGet(rowDoc, "Columns")) - for _, col := range cols { - colDoc, ok := col.(bson.D) - if !ok { - continue - } - if ctx := findEntityContextInWidgets(colDoc, "Widgets", widgetName, currentEntity); ctx != "" { - return ctx - } - } - } - } - tabPages := bsonnav.DGetArrayElements(bsonnav.DGet(wDoc, "TabPages")) - for _, tp := range tabPages { - tpDoc, ok := tp.(bson.D) - if !ok { - continue - } - if ctx := findEntityContextInWidgets(tpDoc, "Widgets", widgetName, currentEntity); ctx != "" { - return ctx - } - } - if controlBar := bsonnav.DGetDoc(wDoc, "ControlBar"); controlBar != nil { - if ctx := findEntityContextInWidgets(controlBar, "Items", widgetName, currentEntity); ctx != "" { - return ctx - } - } - if strings.Contains(typeName, "CustomWidget") { - if obj := bsonnav.DGetDoc(wDoc, "Object"); obj != nil { - props := bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) - for _, prop := range props { - propDoc, ok := prop.(bson.D) - if !ok { - continue - } - valDoc := bsonnav.DGetDoc(propDoc, "Value") - if valDoc == nil { - continue - } - if ctx := findEntityContextInWidgets(valDoc, "Widgets", widgetName, currentEntity); ctx != "" { - return ctx - } - // One level deeper: an object-list item (a DataGrid2 column, an - // Accordion group, a PopupMenu item) is a WidgetObject of its own, - // and its widgets hang off ITS properties — Objects[].Properties[] - // .Value.Widgets. The loop above only reaches the grid's own widget - // properties, so a customContent cell was invisible to this walk and - // everything inside it reported no enclosing entity. That is the - // same descent findInWidgetChildren gained in #834; here it was - // still missing, so ALTER PAGE could FIND those widgets but built - // their bindings with an empty entity context — an association path - // in ContentParams then landed in the document as a literal - // attribute name (CE1613, #935). - // - // Deliberately keyed on the BSON shape rather than on the schema's - // "columns" property key: the same nesting carries every pluggable - // object list, and reading the key would tie the walk to one widget. - for _, item := range bsonnav.DGetArrayElements(bsonnav.DGet(valDoc, "Objects")) { - itemDoc, ok := item.(bson.D) - if !ok { - continue - } - for _, itemProp := range bsonnav.DGetArrayElements(bsonnav.DGet(itemDoc, "Properties")) { - itemPropDoc, ok := itemProp.(bson.D) - if !ok { - continue - } - itemValDoc := bsonnav.DGetDoc(itemPropDoc, "Value") - if itemValDoc == nil { - continue - } - if ctx := findEntityContextInWidgets(itemValDoc, "Widgets", widgetName, currentEntity); ctx != "" { - return ctx - } - } - } - } - } - } - return "" -} - // findNearestDataSourceDoc returns the "DataSource" sub-document of the NEAREST // container enclosing widgetName that declares one, and whether widgetName was // found at all. Unlike findEnclosingEntityContext — which resolves to an entity @@ -1872,7 +1826,7 @@ func findNearestDSInWidgets(parentDoc bson.D, key string, widgetName string, cur return curDS, true } childDS := curDS - if ds := bsonnav.DGetDoc(wDoc, "DataSource"); ds != nil { + if ds := widgetOwnDataSourceDoc(wDoc); ds != nil { childDS = ds } if ds, found := findNearestDSInChildren(wDoc, widgetName, childDS); found { @@ -1928,9 +1882,35 @@ func findNearestDSInChildren(wDoc bson.D, widgetName string, curDS bson.D) (bson if !ok { continue } - if valDoc := bsonnav.DGetDoc(propDoc, "Value"); valDoc != nil { - if ds, found := findNearestDSInWidgets(valDoc, "Widgets", widgetName, curDS); found { - return ds, true + valDoc := bsonnav.DGetDoc(propDoc, "Value") + if valDoc == nil { + continue + } + if ds, found := findNearestDSInWidgets(valDoc, "Widgets", widgetName, curDS); found { + return ds, true + } + // One level deeper: an object-list item (a DataGrid 2 column, an + // Accordion group) keeps its widgets at Objects[].Properties[] + // .Value.Widgets. The entity walk gained this descent in #935 and + // this one did not, so a widget in a customContent cell under a + // FLOW-sourced grid was never reached at all. + for _, item := range bsonnav.DGetArrayElements(bsonnav.DGet(valDoc, "Objects")) { + itemDoc, ok := item.(bson.D) + if !ok { + continue + } + for _, itemProp := range bsonnav.DGetArrayElements(bsonnav.DGet(itemDoc, "Properties")) { + itemPropDoc, ok := itemProp.(bson.D) + if !ok { + continue + } + itemValDoc := bsonnav.DGetDoc(itemPropDoc, "Value") + if itemValDoc == nil { + continue + } + if ds, found := findNearestDSInWidgets(itemValDoc, "Widgets", widgetName, curDS); found { + return ds, true + } } } } @@ -1939,10 +1919,6 @@ func findNearestDSInChildren(wDoc bson.D, widgetName string, curDS bson.D) (bson return nil, false } -func extractEntityFromDataSource(wDoc bson.D) string { - return entityFromEntityRef(bsonnav.DGetDoc(bsonnav.DGetDoc(wDoc, "DataSource"), "EntityRef")) -} - // entityFromEntityRef resolves a datasource's EntityRef to an entity name, // covering both shapes Mendix stores. Shared by the plain-widget and pluggable // readers: the pluggable one handled only the direct form, so an diff --git a/mdl/backend/pagemutator/mutator_selection_source_test.go b/mdl/backend/pagemutator/mutator_selection_source_test.go new file mode 100644 index 000000000..dc718900f --- /dev/null +++ b/mdl/backend/pagemutator/mutator_selection_source_test.go @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "go.mongodb.org/mongo-driver/bson" +) + +// --------------------------------------------------------------------------- +// mendixlabs/mxcli#1076: the entity walk and the datasource walk each knew a +// different subset of the ten Forms$*Source kinds, so ALTER PAGE built the +// replacement widget's bindings in the wrong scope: +// +// - Forms$ListenTargetSource (a data view bound `datasource: selection lv`) +// carries no EntityRef at all — only the listen target's NAME — so the walk +// left the context at the OUTER data view and the binding re-scoped to it +// (CE1613 "the selected attribute no longer exists"). +// - a PLUGGABLE list (Gallery, DataGrid 2) keeps its source under +// Object.Properties[datasource].Value.DataSource, which the flow reader +// never looked at, so a microflow/nanoflow-sourced gallery contributed +// nothing and the binding was written unbound (CE0402). +// +// Both are the FINDINGS #55 / #935 failure again: the walk must resolve EVERY +// source kind, and a nearer source that resolves to nothing must SHADOW the +// outer entity rather than let it be inherited. +// --------------------------------------------------------------------------- + +// selectionDataView builds `dataview (datasource: selection )`. +func selectionDataView(name, target string, children ...bson.D) bson.D { + arr := bson.A{int32(2)} + for _, c := range children { + arr = append(arr, c) + } + return bson.D{ + {Key: "$Type", Value: "Forms$DataView"}, + {Key: "Name", Value: name}, + {Key: "Widgets", Value: arr}, + {Key: "DataSource", Value: bson.D{ + {Key: "$Type", Value: "Forms$ListenTargetSource"}, + {Key: "ListenTarget", Value: target}, + }}, + } +} + +// dataViewOn builds an ordinary entity-bound data view. +func dataViewOn(name, entity string, children ...bson.D) bson.D { + arr := bson.A{int32(2)} + for _, c := range children { + arr = append(arr, c) + } + return bson.D{ + {Key: "$Type", Value: "Forms$DataView"}, + {Key: "Name", Value: name}, + {Key: "Widgets", Value: arr}, + {Key: "DataSource", Value: bson.D{ + {Key: "$Type", Value: "Forms$DataViewSource"}, + {Key: "EntityRef", Value: directEntityRef(entity)}, + }}, + } +} + +// pluggableList builds a Gallery-shaped CustomWidget: a `datasource` property +// holding the given DataSource document, and a `content` property holding the +// template widgets. This is where a pluggable widget keeps its source — NOT at +// the widget's top level, which is the only place the flow reader looked. +func pluggableList(name string, dataSource bson.D, content bson.A) bson.D { + dsID := idBin(0x40) + contentID := idBin(0x41) + return bson.D{ + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Name", Value: name}, + {Key: "Type", Value: bson.D{ + {Key: "ObjectType", Value: bson.D{ + {Key: "PropertyTypes", Value: bson.A{ + int32(2), + bson.D{{Key: "$ID", Value: dsID}, {Key: "PropertyKey", Value: "datasource"}}, + bson.D{{Key: "$ID", Value: contentID}, {Key: "PropertyKey", Value: "content"}}, + }}, + }}, + }}, + {Key: "Object", Value: bson.D{ + {Key: "Properties", Value: bson.A{ + int32(2), + bson.D{ + {Key: "TypePointer", Value: dsID}, + {Key: "Value", Value: bson.D{{Key: "DataSource", Value: dataSource}}}, + }, + bson.D{ + {Key: "TypePointer", Value: contentID}, + {Key: "Value", Value: bson.D{{Key: "Widgets", Value: content}}}, + }, + }}, + }}, + } +} + +func microflowSource(qn string) bson.D { + return bson.D{ + {Key: "$Type", Value: "Forms$MicroflowSource"}, + {Key: "MicroflowSettings", Value: bson.D{ + {Key: "$Type", Value: "Forms$MicroflowSettings"}, + {Key: "Microflow", Value: qn}, + }}, + } +} + +func databaseSource(entity string) bson.D { + return bson.D{ + {Key: "$Type", Value: "Forms$CustomWidgetXPathSource"}, + {Key: "EntityRef", Value: directEntityRef(entity)}, + } +} + +// The reported repro: a selection data view nested inside a page-level data +// view. Its children belong to the LISTEN TARGET's entity (the association +// list's destination), never to the outer data view's. +func TestEnclosingEntity_SelectionSource(t *testing.T) { + row := makeWidget("txtRow", "Forms$DynamicText") + lv := makeAssociationListView("lvVersions", "Bug.Version_Dashboard", "Bug.DashboardVersion", row) + sel := selectionDataView("dvSelected", "lvVersions", makeWidget("txtPreview", "Forms$DynamicText")) + outer := dataViewOn("dvDashboard", "Bug.Dashboard", lv, sel) + m := &Mutator{rawData: pageWith(outer), widgetFinder: findBsonWidget} + + // REPLACE / INSERT BEFORE|AFTER: the target's enclosing scope. + if got := m.EnclosingEntity("txtPreview"); got != "Bug.DashboardVersion" { + t.Errorf("EnclosingEntity(txtPreview) = %q, want Bug.DashboardVersion", got) + } + // INSERT INTO the selection data view itself. + if got := m.EnclosingEntityForChildren("dvSelected"); got != "Bug.DashboardVersion" { + t.Errorf("EnclosingEntityForChildren(dvSelected) = %q, want Bug.DashboardVersion", got) + } + // Control: the outer data view still governs its own direct children, so a + // fix cannot pass by returning the listen target's entity everywhere. + if got := m.EnclosingEntity("lvVersions"); got != "Bug.Dashboard" { + t.Errorf("EnclosingEntity(lvVersions) = %q, want Bug.Dashboard", got) + } +} + +// A selection source may listen to a PLUGGABLE list, whose own source lives in +// its Object.Properties rather than at the widget's top level. +func TestEnclosingEntity_SelectionOfPluggableList(t *testing.T) { + gallery := pluggableList("galItems", databaseSource("Bug.Item"), bson.A{ + int32(2), + makeWidget("txtCard", "Forms$DynamicText"), + }) + sel := selectionDataView("dvSel", "galItems", makeWidget("txtSel", "Forms$DynamicText")) + m := &Mutator{rawData: pageWith(gallery, sel), widgetFinder: findBsonWidget} + + if got := m.EnclosingEntity("txtSel"); got != "Bug.Item" { + t.Errorf("EnclosingEntity(txtSel) = %q, want Bug.Item", got) + } +} + +// A selection source whose target is a FLOW-sourced list has no entity anywhere +// in the page: the mutator must hand the executor the flow's qualified name so +// its RETURN type can be resolved through the model. +func TestEnclosingDataSourceFlow_SelectionOfFlowSourcedList(t *testing.T) { + lv := makeMicroflowListView("lvRows", "Bug.DS_Rows", makeWidget("txtRow", "Forms$DynamicText")) + sel := selectionDataView("dvSel", "lvRows", makeWidget("txtSel", "Forms$DynamicText")) + m := &Mutator{rawData: pageWith(lv, sel), widgetFinder: findBsonWidget} + + if mf, nf := m.EnclosingDataSourceFlow("txtSel", false); mf != "Bug.DS_Rows" || nf != "" { + t.Errorf("EnclosingDataSourceFlow(txtSel) = (%q,%q), want (Bug.DS_Rows, )", mf, nf) + } + if mf, _ := m.EnclosingDataSourceFlow("dvSel", true); mf != "Bug.DS_Rows" { + t.Errorf("EnclosingDataSourceFlow(dvSel, forChildren) = %q, want Bug.DS_Rows", mf) + } +} + +// A pluggable list bound to a microflow: the flow reader looked only at the +// widget's top-level DataSource key, so a Gallery sourced by a microflow +// reported no flow and the replacement widget was written unbound. +func TestEnclosingDataSourceFlow_PluggableSource(t *testing.T) { + gallery := pluggableList("galMf", microflowSource("Bug.DS_Items"), bson.A{ + int32(2), + makeContainerWidget("ctnCard", makeWidget("txtName", "Forms$DynamicText")), + }) + m := &Mutator{rawData: pageWith(gallery), widgetFinder: findBsonWidget} + + if mf, nf := m.EnclosingDataSourceFlow("txtName", false); mf != "Bug.DS_Items" || nf != "" { + t.Errorf("EnclosingDataSourceFlow(txtName) = (%q,%q), want (Bug.DS_Items, )", mf, nf) + } + if mf, _ := m.EnclosingDataSourceFlow("galMf", true); mf != "Bug.DS_Items" { + t.Errorf("EnclosingDataSourceFlow(galMf, forChildren) = %q, want Bug.DS_Items", mf) + } +} + +// The same gap one level deeper: a widget inside a DataGrid 2 customContent +// cell, under a flow-sourced grid. The datasource walk never descended into an +// object-list item's own widgets — the descent the entity walk gained in #935. +func TestEnclosingDataSourceFlow_InsideCustomContentColumn(t *testing.T) { + grid := buildDataGridWithCustomContentColumn(nil, cellContainer()) + // Swap the column grid's datasource for a microflow source. + grid = withPluggableDataSource(t, grid, microflowSource("Bug.DS_Orders")) + m := &Mutator{rawData: pageWith(grid), widgetFinder: findBsonWidget} + + if mf, _ := m.EnclosingDataSourceFlow("txtCust", false); mf != "Bug.DS_Orders" { + t.Errorf("EnclosingDataSourceFlow(txtCust) = %q, want Bug.DS_Orders", mf) + } +} + +// withPluggableDataSource replaces the `datasource` property's DataSource +// document on a pluggable widget built by buildDataGridWithCustomContentColumn. +func withPluggableDataSource(t *testing.T, widget bson.D, ds bson.D) bson.D { + t.Helper() + keys := buildPropKeyMap(widget) + obj := bsonnav.DGetDoc(widget, "Object") + for _, p := range bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) { + pd, ok := p.(bson.D) + if !ok { + continue + } + if keys[bsonnav.ExtractBinaryIDFromDoc(bsonnav.DGet(pd, "TypePointer"))] != "datasource" { + continue + } + for i, kv := range pd { + if kv.Key == "Value" { + pd[i].Value = bson.D{{Key: "DataSource", Value: ds}} + } + } + return widget + } + t.Fatal("no datasource property on the test widget") + return widget +} + +// A flow-sourced list nested inside an entity-bound data view must SHADOW the +// outer entity: inheriting it is how the wrong binding got written in the first +// place. The entity is empty and the flow is reported instead. +func TestEnclosingEntity_FlowSourceShadowsOuterEntity(t *testing.T) { + lv := makeMicroflowListView("lvRows", "Bug.DS_Rows", makeWidget("txtRow", "Forms$DynamicText")) + outer := dataViewOn("dvWeek", "Bug.Week", lv) + m := &Mutator{rawData: pageWith(outer), widgetFinder: findBsonWidget} + + if got := m.EnclosingEntity("txtRow"); got != "" { + t.Errorf("EnclosingEntity(txtRow) = %q, want \"\" (the microflow list shadows Bug.Week)", got) + } + if mf, _ := m.EnclosingDataSourceFlow("txtRow", false); mf != "Bug.DS_Rows" { + t.Errorf("EnclosingDataSourceFlow(txtRow) = %q, want Bug.DS_Rows", mf) + } +} + +// Robustness controls: a selection source naming a widget that does not exist, +// and two data views listening to each other, must terminate and report no +// scope — never the outer entity, and never a stack overflow. +func TestSelectionSource_DanglingAndCyclicTargets(t *testing.T) { + dangling := selectionDataView("dvSel", "noSuchWidget", makeWidget("txtSel", "Forms$DynamicText")) + outer := dataViewOn("dvOuter", "Bug.Week", dangling) + m := &Mutator{rawData: pageWith(outer), widgetFinder: findBsonWidget} + if got := m.EnclosingEntity("txtSel"); got != "" { + t.Errorf("dangling listen target: EnclosingEntity(txtSel) = %q, want \"\"", got) + } + + a := selectionDataView("dvA", "dvB", makeWidget("txtA", "Forms$DynamicText")) + b := selectionDataView("dvB", "dvA") + m2 := &Mutator{rawData: pageWith(a, b), widgetFinder: findBsonWidget} + if got := m2.EnclosingEntity("txtA"); got != "" { + t.Errorf("cyclic listen targets: EnclosingEntity(txtA) = %q, want \"\"", got) + } +} From 6fec7ad239d8f51dc73ab60cf6f7d18546c1cffc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 20:44:40 +0000 Subject: [PATCH 04/18] docs(microflow): audit what a rewrite loses, across 342 microflows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `describe microflow` -> `exec` is the documented copy operation and does not round-trip. Audited across 342 microflows in 4 projects (TestApp, CapTrack, RestLab, Ledger; 11.14.0), two properties change with nothing reporting it — `mxcli check` is quiet and mxbuild is quiet, because the model is valid either way: - ApplyEntityAccess true -> false (16/342, hardcoded in both writers) - ShowMessageAction.Blocking true -> false (16 microflows, a DESCRIBE gap) The first is a security setting. "Apply entity access" makes a microflow run under the current user's access rules; turning it off WIDENS what it can read and write, and the only visible consequence is different behaviour for a user whose role should have been constraining them. They need different fixes, which is why the report separates them by cause rather than by symptom. ApplyEntityAccess is hardcoded false in both writers and `microflows.Microflow` has no field for it — while `microflows.Rule` carries it correctly on both engines, so this was never an unknown-property gap. Blocking is carried perfectly end to end by both engines; MDL simply has no `blocking` keyword, so the text round trip drops it. Three claims in the first draft did not survive checking, and the report says so rather than quietly omitting them: - ConcurrenyErrorMessage looked like 58 lost translations. Every one holds Text: "" — an empty entry versus no entry. Demoted out of the severity table; the writer's hardcode is still a hole, just not a demonstrated bug. - ActionActivity.Caption "Activity" -> "" happens only where AutoGenerateCaption is true. A user-set caption survives exactly. - CaseValues gaining Microflows$NoCase is mxcli FIXING an older shape — Mendix 10 rejects a bare marker as CE0079/CE0773. The control is StableId: preserved 342/342, exactly as ADR-0008 claims. The method normalises element $IDs away, so without a property known to be carried there would be no way to tell a real loss from the normalisation being too aggressive. Also lands the tooling under scripts/microflow-roundtrip-audit/, so the measurement can be repeated after a fix instead of described. Its README records the three distinctions that decide whether a diff matters — layout vs behaviour, empty vs absent, hardcoded-writer vs unspellable-DESCRIBE — each of which already caught something here. No code changes: this is the audit, not the repair. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .../microflow-rewrite-property-audit.md | 144 ++++++++++++++++++ scripts/microflow-roundtrip-audit/README.md | 48 ++++++ scripts/microflow-roundtrip-audit/classify.py | 75 +++++++++ .../microflow-roundtrip-audit/roundtrip.sh | 34 +++++ 5 files changed, 302 insertions(+) create mode 100644 docs/12-bug-reports/microflow-rewrite-property-audit.md create mode 100644 scripts/microflow-roundtrip-audit/README.md create mode 100644 scripts/microflow-roundtrip-audit/classify.py create mode 100755 scripts/microflow-roundtrip-audit/roundtrip.sh diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index adb37ab05..68d90ce2a 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -90,3 +90,4 @@ {"area": "mdl-backend", "date": "2026-09-10", "symptom": "`mxcli diff-local` on an MPR v2 project fails with `Error: mprcontents directory not found` while mprcontents/ exists and is populated; `MXCLI_ENGINE=legacy` works", "cause": "The modelsdk engine (the default) never overrode `Backend.ContentsDir()`, so it fell through to the generated `unimplemented` stub and returned \"\". diff-local reads \"\" as 'not a v2 project'.", "file": "mdl/backend/modelsdk/backend.go", "insight": "gen_unimplemented.go's promise that an unoverridden method 'fails loudly rather than silently dropping data' is CONDITIONAL on the method having an error to fail through: the generated body is `errUnimplemented` only when a result is `error`, a panic when there are no results at all, and a silent `var r0 T; return r0` otherwise. ContentsDir is in the third bucket and its zero value is a MEANINGFUL in-band answer (\"\" == MPR v1), so the missing implementation was indistinguishable from a v1 project rather than looking like a bug. The detectable signature was the contradiction between two questions the same command asks: Version() (implemented) says 2, ContentsDir() says v1. Guard added in mdl/backend/modelsdk/unimplemented_silent_test.go \u2014 reflect over FullBackend for error-less methods, go/parser the package for methods actually declared on *Backend, since reflection cannot tell a promoted method from an override (Go synthesises a wrapper named (*Backend).X for both). It immediately found a second one, InvalidateCache (a latent panic, no caller today).", "refs": ["mendixlabs/mxcli#1080"]} {"area": "mdl/backend", "date": "2026-09-10", "symptom": "SOAP `call web service` writes a document mxbuild accepts and Studio Pro would not have written. A SEND MAPPING is silently DROPPED by both engines \u2014 `send mapping Mod.Export` parses, `mxcli check` passes, `exec` reports success, and nothing in the stored action references the mapping. Operation ARGUMENTS are dropped the same way", "cause": "sdk/mpr.serializeWebServiceCallAction was written without a Studio Pro reference and hardcodes five things it cannot know, and the codec engine's new writer reproduced it deliberately for parity. Measured against three Studio Pro-authored calls in ako/TestApp (Mendix 11.14.0, Clients.GetOrders / GetCustomerOrders / SaveOrder): ServiceName is the WSDL SERVICE name (\"OrdersWS\") not the local part of the imported service's qualified name (\"OrderSoapClient\"); ImportMappingCall.ContentType is \"Xml\" for a SOAP import mapping, not \"Json\"; Range.SingleObject follows cardinality (false for a list) rather than being always true; VariableType is the real result type (DataTypes$ObjectType with an Entity, DataTypes$BooleanType) rather than always DataTypes$VoidType; and a send mapping is Microflows$MappingRequestHandling {ContentType, MappingId, MappingVariableName}. Arguments live in RequestBodyHandling.ParameterMappings as Microflows$WebServiceOperationSimpleParameterMapping entries keyed by an escaped ParameterPath (\"http%3A//www.example.com/:GetOrder|OrderId\")", "file": "`sdk/mpr/writer_microflow_actions.go` (serializeWebServiceCallAction), `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**A guessed type name in a comment becomes a permanent refusal.** Legacy refused send mappings citing `Mendix$AdvancedRequestHandling`, said it 'requires a Studio Pro-generated example to determine the correct type storage name', and that refusal then shipped for as long as nobody went looking. The real type is `Microflows$MappingRequestHandling` \u2014 which THIS CODEBASE ALREADY WRITES for REST result/request handling \u2014 and the guessed name occurs in none of the three reference documents. The lesson is not about SOAP: when a writer refuses because a storage name is unknown, check whether a sibling feature already writes it before treating the refusal as a standing constraint. **Second, and the reason this was found at all: 'no reference exists' is a claim about where you looked.** The parity work asserted that no Studio Pro-authored SOAP document existed to pin against and used that to justify mirroring legacy; one existed in a separate repo the whole time (ako/TestApp, which carries both a consumed client and a published service). Byte-parity with what ships is a legitimate goal for a change scoped to stopping a silent drop \u2014 it is NOT evidence the shape is right, and conflating the two is how six defects got a passing test. Where a reference project exists, name it in the code so the next reader does not repeat the search", "refs": []} {"area": "mdl/backend", "date": "2026-09-11", "symptom": "Making SOAP calls describe structurally instead of as base64 silently rewrote Studio Pro's own documents: a describe -> exec round trip over ako/TestApp flipped `Range.SingleObject` false -> true with no error at all, and turned Clients.SaveOrder's `DataTypes$BooleanType` result into VoidType, which mxbuild reported as `[CE0366]` + `[CE6011]`", "cause": "`webServiceActionRequiresRawBSON` admitted keys BY NAME. That was sound while the supported set was the only nine keys the writer emitted \u2014 but a real call carries FIFTEEN, and six of the new ones are boilerplate mxcli writes at ONE fixed value. Admitting `HttpConfiguration`, `RequestHeaderHandling`, `IsValidationRequired`, `ProxyConfiguration`, `RequestProxyType` and `NewResultHandling` by name meant any call configured beyond mxcli's defaults would be normalised on the next exec", "file": "`mdl/backend/modelsdk/microflow_read_actions.go`, `sdk/mpr/parser_microflow_actions.go` (webServiceActionRequiresRawBSON and its value predicates)", "insight": "**The question a raw-fallback gate answers is not 'do I know this key' but 'would writing this back produce the same document'.** The two coincide only while the writer emits exactly the supported set; the moment a feature lands that widens what is representable, the by-name test starts approving documents it cannot reproduce \u2014 and the loss is invisible, because the result is a VALID model that differs from the user's. **The round trip is the only thing that catches it**: unit tests on the new feature all passed, `mx check` on the newly-written calls was 0 errors, and the regression only appeared when describe -> exec was run over the REFERENCE documents and the BSON diffed (`mxcli bson dump --type microflow --object`, ids normalised away). Two of the five diffs that surfaced were pre-existing microflow describe drift (NoCase case values, bezier control vectors) and unrelated \u2014 worth separating before blaming the change. Fixing it also SHRANK the feature's reach honestly: Studio Pro's own SOAP calls keep the raw form until `Range.SingleObject` is explained, and only mxcli-authored calls describe structurally. **A result type can come from somewhere MDL cannot see** \u2014 SaveOrder's Boolean is the WSDL operation's return type, not an import mapping's entity \u2014 so 'binds a result' does not imply 'derivable'", "refs": []} +{"area": "mdl/backend", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec` \u2014 the documented copy operation \u2014 silently turns OFF a microflow's \"apply entity access\". Measured across 342 microflows in 4 projects (11.14.0): every microflow storing `ApplyEntityAccess: true` came back `false` (16/342, all 4 distinct Administration documents). A blocking `show message` also becomes non-blocking (16 microflows). `mxcli check` and mxbuild are both silent \u2014 the model is valid either way", "cause": "TWO causes wearing one symptom. (1) `ApplyEntityAccess` is HARDCODED false in both writers (`mdl/backend/modelsdk/microflow_write.go` SetApplyEntityAccess(false), `sdk/mpr/writer_microflow.go` {Key:\"ApplyEntityAccess\", Value:false}) and `microflows.Microflow` has no field for it, so the read side drops it first. (2) `ShowMessageAction.Blocking` is carried correctly end to end on BOTH engines \u2014 the loss is in DESCRIBE, which has no `blocking` keyword to emit, so the re-parse sets false", "file": "`mdl/backend/modelsdk/microflow_write.go`, `sdk/mpr/writer_microflow.go`, `sdk/microflows/microflows.go` (Microflow struct), `mdl/executor/cmd_microflows_format_action.go` (show message)", "insight": "**A round-trip audit needs a preservation control, or it cannot tell a bug from its own blind spot.** Here it was `StableId`: 342/342 preserved, exactly as ADR-0008 claims \u2014 a method that normalised ids away too aggressively would have reported that as churn, and everything else with it. **Separate `hardcoded in the writer` from `unspellable in DESCRIBE`**: they look identical in a before/after diff and need completely different fixes (model plumbing vs grammar), and `Blocking` proves a property can be perfectly carried by both engines and still be lost by the text round trip. **The same property handled two ways in one codebase is the tell**: `microflows.Rule` carries ApplyEntityAccess correctly (`rule_write.go`, `parser_rule.go`) while `microflows.Microflow` does not \u2014 so this was never an unknown-property gap, just an unfinished one. **Check what an 'empty' value actually holds before calling it a loss**: `ConcurrenyErrorMessage` looked like 58 lost translations and every single one had `Text: \"\"`, i.e. an empty entry versus no entry. Two more traps worth knowing: `ActionActivity.Caption` changes ONLY where `AutoGenerateCaption` is true (a user-set caption survives), and ~90% of the 417-line diff on a real microflow is layout \u2014 bezier vectors, sizes, connection indices \u2014 which buries the two lines that matter", "refs": []} diff --git a/docs/12-bug-reports/microflow-rewrite-property-audit.md b/docs/12-bug-reports/microflow-rewrite-property-audit.md new file mode 100644 index 000000000..fe1496bfb --- /dev/null +++ b/docs/12-bug-reports/microflow-rewrite-property-audit.md @@ -0,0 +1,144 @@ +# Bug Report: what a microflow rewrite loses — a property-by-property audit + +## Summary + +`describe microflow` → `exec` is the documented copy operation, and it does not +round-trip. Audited across **342 microflows in 4 projects** (TestApp, CapTrack, +RestLab, Ledger; Mendix 11.14.0), **two** properties change with no error from +`mxcli check` or from mxbuild, and one of them is a security setting: + +| property | class | affected | effect | +|---|---|---|---| +| `ApplyEntityAccess` | **writer hardcodes `false`** | 16 / 342 | a microflow that ran **under entity access** starts running without it | +| `ShowMessageAction.Blocking` | **DESCRIBE cannot spell it** | 16 microflows | a **blocking** message box becomes non-blocking | + +Everything else that differs is layout or an empty-vs-absent artefact — listed in +full below, because "we checked and the rest is fine" is worth nothing unless the +checking is shown. + +## Impact + +`ApplyEntityAccess` is the serious one. Mendix's "apply entity access" makes a +microflow run under the current user's entity access rules rather than with full +access. Turning it off **widens** what the microflow can read and write, and +nothing downstream reports it: the model is valid, the app builds, and the +behaviour only differs for a user whose role should have been constraining them. + +The path that triggers it is the one the docs recommend. CLAUDE.md names +`describe` → rename → `exec` as *the* copy operation (there is deliberately no +`COPY DOCUMENT` verb), and `CREATE OR REPLACE MICROFLOW` rebuilds from the +statement, so anything the statement does not restate is gone. + +## Reproduction + +```bash +cp -r /tmp/audit && cd /tmp/audit +mxcli -p App.mpr -c "describe microflow Administration.SaveNewAccount" > rt.mdl +MXCLI_ALWAYS_WRITE=1 mxcli exec rt.mdl -p App.mpr + +mxcli bson dump -p App.mpr --type microflow --object Administration.SaveNewAccount \ + | grep -A1 ApplyEntityAccess +``` + +`Administration.SaveNewAccount` ships with every blank Mendix app, stores +`ApplyEntityAccess: true`, and comes back `false`. `MXCLI_ALWAYS_WRITE=1` is only +there to defeat idempotent-write elision so the write definitely lands. + +## Root cause + +Two different causes wearing one symptom, which is why they need two fixes: + +**1. `ApplyEntityAccess` — hardcoded in both writers.** + +```go +// mdl/backend/modelsdk/microflow_write.go:208 +out.SetApplyEntityAccess(false) + +// sdk/mpr/writer_microflow.go:96 +{Key: "ApplyEntityAccess", Value: false}, +``` + +Not an unknown property: `microflows.Rule` carries it correctly on both engines +(`rule_write.go:104`, `parser_rule.go:52`). A **rule** keeps its setting and a +**microflow** does not, in the same codebase. `microflows.Microflow` has no field +for it at all, so the read side drops it before the writer is reached. + +**2. `ShowMessageAction.Blocking` — a DESCRIBE gap, not a writer gap.** + +`Blocking` is carried correctly through the model on both engines +(`microflow_read_actions.go:263`, `microflow_write.go:794`, +`writer_microflow_actions.go:511`). The loss is in the middle: MDL has no +`blocking` keyword, so `describe` emits + +```mdl +show message 'The password has been updated.' type Information; +``` + +and the re-parse sets `Blocking: false`. Anything rewriting the microflow from +**stored BSON** keeps it; only the round trip through MDL text loses it. + +## Method, and what it is worth + +For every microflow in each project: dump the stored BSON, `describe` it, `exec` +that with `MXCLI_ALWAYS_WRITE=1`, dump again, and diff with element `$ID`s +normalised away and list order ignored (so reordering does not masquerade as a +value change). 341 of 342 were confirmed rewritten (`Replaced microflow` in the +exec log); the one exception is recorded in `exec_failures.txt`. + +**The control is `StableId`: preserved in 342 / 342.** ADR-0008 says it is +carried rather than re-minted, and a method that could not tell preservation from +loss would have reported it as churn. It did not. + +Two honest limitations: + +- **All 16 microflows with `ApplyEntityAccess: true` are the same 4 + Administration microflows**, present in all four projects because every blank + app ships that Marketplace module. No *user-written* microflow in this corpus + sets the flag. So "16 / 342" is not a base rate — what is established is that + **every microflow observed to carry the flag lost it, 4 of 4 distinct + documents**, which follows from the hardcode regardless of sample size. +- The corpus is four projects on one Mendix version. A property no document here + sets cannot be cleared by this audit — see `ConcurrenyErrorMessage` below. + +## Everything else that differs, and why it is not in the table + +| what changes | count | why it is benign | +|---|---|---| +| `ConcurrenyErrorMessage` loses its `Texts$Translation` entry | 20 / 58 stored | **the `Text` is `""` in all 58** — an empty translation entry versus no entry. The writer does hardcode an empty `Texts$Text`, so a real message *would* be lost, but no document in this corpus has one, so nothing here shows it. Same for the hardcoded `ConcurrencyErrorMicroflow: ""` (all 342 already `""`). Both are inert unless `AllowConcurrentExecution` is false, which nothing here is | +| `BezierCurve` Origin/DestinationControlVector → `"0;0"` | 87 / 56 | connector curvature; layout only | +| `ActionActivity.Caption` `"Activity"` → `""` | 74 | **only where `AutoGenerateCaption: true`** — a user-set caption (`"Save password"`, auto=false) is preserved exactly | +| `Size`, `RelativeMiddlePoint`, `Origin/DestinationConnectionIndex` | 16–28 | layout only | +| `CaseValues` gains `Microflows$NoCase` | 37 | required from Mendix 10; a bare `CaseValues: [marker]` is CE0079/CE0773. This is mxcli **fixing** an older shape | +| `Attribute`, `XpathConstraint`, `LocalVariable`, `Documentation` `""` → absent | 21–30 | empty-vs-absent; no value lost. A non-empty value survives (`System.User.WebServiceUser` is kept while its `""` sibling vanishes) | +| `CloseFormAction.NumberOfPages` absent → `1` | 16 | mxcli adds a key Studio Pro omits; the value is the default | +| `Documentation` `\r\n` → `\n` | 1 | CRLF normalisation | + +None of these changes behaviour. They do explain why a one-line edit produces a +**417-line diff** on a real microflow, which is its own reviewability problem and +worth a separate issue. + +## Suggested fixes, in severity order + +1. **`ApplyEntityAccess`** — add the field to `microflows.Microflow`, read it on + both engines, and give MDL a way to say it. The rule path is the precedent to + copy. Until the syntax exists, the safe interim is **guard-don't-drop** + (ADR-0005): refuse to rewrite a microflow whose stored flag is `true`, rather + than silently clearing it. +2. **`ShowMessageAction.Blocking`** — a `blocking` modifier on `show message`. + The model already carries it end to end; only the grammar, visitor and + describe formatter are missing. +3. **`ConcurrenyErrorMessage` / `ConcurrencyErrorMicroflow`** — carry both on the + model rather than hardcoding. No loss is demonstrated here, so this is closing + a hole rather than fixing a bug, and it needs a project that sets them before + the shape can be pinned. (Note Mendix's own spelling: `ConcurrenyErrorMessage` + is one `c` short of "Concurrency", unlike its sibling.) + +The first two need MDL syntax or model plumbing; neither is a one-line change. + +## Environment + +- mxcli: branch `claude/mxcli-unit-test-perf-n7ggx8`, commit `7261dfdd` +- Mendix: 11.14.0 (mxbuild 11.14.0) +- Corpus: ako/TestApp (42), CapTrack (138), RestLab (60), Ledger (102) +- Both engines: the two writers hardcode the same values, so `MXCLI_ENGINE` makes + no difference to any finding here diff --git a/scripts/microflow-roundtrip-audit/README.md b/scripts/microflow-roundtrip-audit/README.md new file mode 100644 index 000000000..64cd892af --- /dev/null +++ b/scripts/microflow-roundtrip-audit/README.md @@ -0,0 +1,48 @@ +# Microflow round-trip audit + +Measures what `describe microflow` → `exec` changes about a microflow — the +documented copy operation, which does not round-trip. Not wired into CI: it +rewrites every microflow in a project and takes minutes, so it is run on demand +against real projects. + +Its first run produced +[`microflow-rewrite-property-audit.md`](../../docs/12-bug-reports/microflow-rewrite-property-audit.md); +re-run it after a fix to show the property has stopped moving. + +## Use + +```bash +make build +scripts/microflow-roundtrip-audit/roundtrip.sh +python3 scripts/microflow-roundtrip-audit/classify.py +``` + +`roundtrip.sh` copies the project first — it never touches the original. For each +microflow it dumps the stored BSON, describes it, re-executes that MDL with +`MXCLI_ALWAYS_WRITE=1` (so idempotent-write elision cannot hide a change), and +dumps again. `/exec_failures.txt` lists any describe output that did not +re-execute, which is a finding in its own right. + +`classify.py` reports what moved, split into the microflow's top-level +properties and its flow graph (`ObjectCollection` / `Flows`) — different concerns, +and the graph is where the layout noise lives. + +## Reading the output + +Three distinctions decide whether a difference matters, and all three have +already caught something: + +- **Layout versus behaviour.** Roughly 90% of a real microflow's diff is bezier + control vectors, sizes, positions and connection indices. They bury the lines + that matter; filter them before drawing conclusions. +- **Empty versus absent.** `""` → key absent is not a loss. `ConcurrenyErrorMessage` + looked like 58 dropped translations and every one held `Text: ""`. +- **Hardcoded-in-the-writer versus unspellable-in-DESCRIBE.** Identical in the + diff, completely different fixes — model plumbing versus grammar. `Blocking` is + carried perfectly by both engines and still lost, because DESCRIBE cannot say it. + +**Keep a preservation control.** `classify.py` normalises element `$ID`s away so +that re-minted ids do not swamp the output; `StableId` is the check that the +normalisation has not gone too far, since ADR-0008 says it is carried and the +audit should therefore report it unchanged (342 / 342 on the first run). A run +that shows `StableId` moving is measuring its own blind spot, not a bug. diff --git a/scripts/microflow-roundtrip-audit/classify.py b/scripts/microflow-roundtrip-audit/classify.py new file mode 100644 index 000000000..3c0cd90bd --- /dev/null +++ b/scripts/microflow-roundtrip-audit/classify.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Classify what a describe -> exec round trip changed, per microflow. + +Top-level document properties are reported separately from the flow graph +(ObjectCollection / Flows): they are different concerns — one is the +microflow's settings, the other its shape. +""" +import json, os, sys, collections + +GRAPH_KEYS = {"ObjectCollection", "Flows"} + + +def to_plain(v): + """mxcli bson dump emits [{Key,Value},...]; normalise to dicts, ids away.""" + if isinstance(v, list): + if v and all(isinstance(e, dict) and set(e) == {"Key", "Value"} for e in v): + return {e["Key"]: to_plain(e["Value"]) for e in v if e["Key"] != "$ID"} + return [to_plain(e) for e in v] + if isinstance(v, dict): + if "Data" in v and "Subtype" in v: + return "" + return {k: to_plain(x) for k, x in v.items() if k != "$ID"} + return v + + +def load(path): + with open(path) as f: + return to_plain(json.loads("[" + f.read())) + + +def main(root): + lost = collections.Counter() # key present before, absent after + changed = collections.Counter() # key present both, value differs + added = collections.Counter() # key absent before, present after + examples = {} + graph_changed = [] + seen = 0 + + for name in sorted(os.listdir(f"{root}/before")): + mf = name[:-5] + before, after = load(f"{root}/before/{name}"), load(f"{root}/after/{name}") + if not isinstance(before, dict) or not isinstance(after, dict): + continue + seen += 1 + for k in set(before) | set(after): + if k in GRAPH_KEYS: + if before.get(k) != after.get(k): + graph_changed.append(mf) + continue + b, a = before.get(k, ""), after.get(k, "") + if b == a: + continue + if a == "": + lost[k] += 1 + elif b == "": + added[k] += 1 + else: + changed[k] += 1 + examples.setdefault(k, (mf, b, a)) + + print(f"== {root}: {seen} microflows") + for label, counter in (("LOST", lost), ("CHANGED", changed), ("ADDED", added)): + if not counter: + continue + print(f"\n {label}") + for k, n in counter.most_common(): + mf, b, a = examples[k] + bs, as_ = json.dumps(b)[:70], json.dumps(a)[:70] + print(f" {k:<28} {n:>3}/{seen} {bs} -> {as_}") + print(f" {'':<28} e.g. {mf}") + print(f"\n flow graph changed in {len(set(graph_changed))}/{seen}") + + +if __name__ == "__main__": + main(sys.argv[1]) diff --git a/scripts/microflow-roundtrip-audit/roundtrip.sh b/scripts/microflow-roundtrip-audit/roundtrip.sh new file mode 100755 index 000000000..7d2839d19 --- /dev/null +++ b/scripts/microflow-roundtrip-audit/roundtrip.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Round-trip audit: for every microflow in a project, describe -> exec -> dump, +# and record what changed. Per-microflow so one failure does not hide the rest. +set -u +if [ $# -ne 3 ]; then + echo "usage: $0 (MXCLI= to override the binary)" >&2 + exit 2 +fi +MXCLI="${MXCLI:-$(cd "$(dirname "$0")/../.." && pwd)/bin/mxcli}" +SRC="$1" # source project dir +MPR="$2" # mpr filename +OUT="$3" # output dir + +rm -rf "$OUT"; mkdir -p "$OUT/before" "$OUT/after" "$OUT/mdl" +WORK="$OUT/work"; rm -rf "$WORK"; cp -r "$SRC" "$WORK" + +mapfile -t MFS < <("$MXCLI" bson dump -p "$WORK/$MPR" --type microflow --list 2>/dev/null \ + | sed -n 's/^ \([A-Za-z0-9_.]*\) (Microflows\$Microflow)$/\1/p') +echo "microflows: ${#MFS[@]}" >&2 + +: > "$OUT/exec_failures.txt" +for mf in "${MFS[@]}"; do + "$MXCLI" bson dump -p "$WORK/$MPR" --type microflow --object "$mf" 2>/dev/null \ + | tail -n +2 > "$OUT/before/$mf.json" + "$MXCLI" -p "$WORK/$MPR" -c "describe microflow $mf" 2>/dev/null > "$OUT/mdl/$mf.mdl" + if ! MXCLI_ALWAYS_WRITE=1 "$MXCLI" exec "$OUT/mdl/$mf.mdl" -p "$WORK/$MPR" >"$OUT/mdl/$mf.log" 2>&1; then + echo "$mf" >> "$OUT/exec_failures.txt" + elif grep -qi "parse error\|Error:" "$OUT/mdl/$mf.log"; then + echo "$mf" >> "$OUT/exec_failures.txt" + fi + "$MXCLI" bson dump -p "$WORK/$MPR" --type microflow --object "$mf" 2>/dev/null \ + | tail -n +2 > "$OUT/after/$mf.json" +done +echo "done: $OUT" >&2 From 8d97e430c4a3805d7b85e973613c84e92dc7df54 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 22:57:08 +0000 Subject: [PATCH 05/18] fix(microflow): stop a rewrite clearing "apply entity access" and "blocking" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two findings from the round-trip audit. Re-running it across the same 342 microflows in 4 projects shows both properties stop moving. ## ApplyEntityAccess — a security setting, cleared on every rewrite "Apply entity access" makes a microflow run under the current user's access rules rather than with full access, so clearing it WIDENS what the microflow may read and write. Nothing reported it: `mxcli check` quiet, mxbuild quiet, model valid either way. The property now exists on `microflows.Microflow`, is read and written by both engines, and the executor PRESERVES a stored value on a rewrite that does not mention it. That last part is the load-bearing half — describe -> exec rebuilds from the STATEMENT, so carrying it through the writers alone would have fixed nothing. The documented COPY operation (describe -> rename -> exec) has nothing to preserve from, so MDL gained a way to say it: `@applyentityaccess` before `create microflow` / `create rule`, `@applyentityaccess(false)` to clear it. Absent-preserves is the same rule `@excluded` (#914) and the doc comment (#1018) already follow, and the AST field is a *bool so silence and an explicit false stay distinguishable. No grammar change — `annotationValue` already accepts a literal. A nanoflow is deliberately excluded: it runs in the client and Mendix stores no such property, so the annotation would parse and do nothing. Rules had the same gap from the other end — `rule_write.go` plumbed the property through while their CREATE path never set it — and are fixed alongside. ## ShowMessageAction.Blocking — carried by both engines, lost in the text A blocking message box halts the client until dismissed. Both engines carried the flag perfectly; MDL had no word for it, so DESCRIBE could not emit it and the re-parse set false. Every layer except the text was correct, which is how it survived. `show message '…' [type T] [objects […]] blocking [on error …]`. BLOCKING is listed in the `keyword` rule so `blocking` remains usable as an ordinary identifier, pinned by a test. ## Verification - The audit re-run: both properties gone from the changed list, 342 microflows, 4 projects. - All three cases: an unstated rewrite preserves a stored true; a copy carrying the annotation creates one with true; `(false)` creates one with false. - Controls: reverting each of the three code changes in turn fails its test with the reported symptom — "ApplyEntityAccess lost on round-trip", "written ApplyEntityAccess = false, want true", and a describe emitting the message with no modifier. Recorded in the report and not fixed here: round-tripping EVERY microflow in a project yields CE0709 "Sequence flow is not accepted by origin or destination". The pre-fix binary reproduces it identically, so it is pre-existing flow-graph drift — easy to misattribute to this change, which is why the control was run. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../skills/mendix/write-microflows/SKILL.md | 5 ++ .../write-microflows/reference/pitfalls.md | 29 +++++++ cmd/mxcli/lsp_completions_gen.go | 1 + cmd/mxcli/syntax/features_microflow.go | 7 ++ docs/01-project/MDL_QUICK_REFERENCE.md | 2 + .../microflow-rewrite-property-audit.md | 67 ++++++++++++++++ .../doctype-tests/02-microflow-examples.mdl | 22 +++++ mdl/ast/ast_microflow.go | 24 ++++++ mdl/backend/modelsdk/microflow.go | 5 ++ .../microflow_roundtrip_flags_test.go | 31 +++++++ mdl/backend/modelsdk/microflow_write.go | 4 +- mdl/executor/apply_entity_access.go | 27 +++++++ mdl/executor/apply_entity_access_test.go | 38 +++++++++ mdl/executor/cmd_microflows_build.go | 6 ++ mdl/executor/cmd_microflows_builder_calls.go | 1 + mdl/executor/cmd_microflows_format_action.go | 8 ++ .../cmd_microflows_format_action_test.go | 22 +++++ mdl/executor/cmd_microflows_show.go | 21 +++++ mdl/executor/cmd_rules_create.go | 6 ++ mdl/grammar/MDLLexer.g4 | 4 + mdl/grammar/domains/MDLMicroflow.g4 | 2 +- mdl/grammar/domains/MDLSettings.g4 | 2 +- .../visitor_apply_entity_access_test.go | 80 +++++++++++++++++++ mdl/visitor/visitor_microflow.go | 35 ++++++++ mdl/visitor/visitor_microflow_actions.go | 6 ++ ...visitor_microflow_actions_blocking_test.go | 51 ++++++++++++ sdk/microflows/microflows.go | 12 +++ sdk/mpr/parser_microflow.go | 6 ++ sdk/mpr/writer_microflow.go | 5 +- sdk/mpr/writer_microflow_flags_test.go | 45 +++++++++++ 31 files changed, 571 insertions(+), 4 deletions(-) create mode 100644 mdl/executor/apply_entity_access.go create mode 100644 mdl/executor/apply_entity_access_test.go create mode 100644 mdl/visitor/visitor_apply_entity_access_test.go create mode 100644 mdl/visitor/visitor_microflow_actions_blocking_test.go create mode 100644 sdk/mpr/writer_microflow_flags_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 95cac460f..d6a305bf6 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -588,3 +588,4 @@ {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A SOAP `call web service` assigning its result was rejected twice over: `[CE0243] \"The mapping used to return a value of type 'Nothing', but now returns a value of type 'Clients.Order'\"` and `[CE0366] \"Cannot store in variable when there is no return value\"` \u2014 on a call whose receive mapping plainly produces an entity", "cause": "Both engines wrote the result handling's VariableType as `DataTypes$VoidType` unconditionally. Void means the call returns nothing, so it contradicts the mapping AND makes the assignment illegal. Studio Pro writes the entity the mapping produces: `DataTypes$ObjectType{Entity: \"Clients.Order\"}`, which is the Entity of the import mapping's ROOT `ImportMappings$ObjectMappingElement`", "file": "`mdl/executor/webservice_names.go` (resolveImportMappingEntity), `sdk/mpr/writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**Fixing one wrong name in a SOAP call reveals the next; work the chain against a real project instead of stopping at the first green.** On ako/TestApp (11.14.0, baseline 0 errors) the SAME one-statement script went StorageLoadException (ReturnValueMapping written as a UUID) -> CE0386 (ServiceName derived instead of read) -> CE0243+CE0366 (VariableType Void) -> CE0178 (operation arguments), four rounds, each error invisible until the previous fix landed. Any of them could have been called 'the' bug. **The enabler each time was reading the referenced DOCUMENT rather than deriving from the statement**: the WSDL service name is in `Description.Services[].Name` of the imported service, the result entity in `Elements[0].Entity` of the import mapping \u2014 both structured, neither needing the embedded WSDL to be parsed. `ListRawUnitsByType` reaches them on both engines with no new backend method, but note it matches the $Type EXACTLY despite the parameter being named typePrefix, and the types are `WebServices$ImportedServiceImpl` and `ImportMappings$ImportMapping` (NOT the `Mappings$` prefix their child elements use). **Every resolver returns \"\" rather than guessing** \u2014 unresolvable, ambiguous, or wrong-shaped all fall back to what shipped, because a made-up name reproduces the same error with different text in it and is harder to recognise. Remaining and measured: CE0178 needs operation arguments, which MDL cannot express at all (callWebServiceStatement has no argument list), and Range.SingleObject differs from Studio Pro with no error yet attached \u2014 the reference mapping roots carry MaxOccurs 1 while the calls carry SingleObject false, so it is not the mapping's cardinality and would be a guess", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "A DESCRIBE-side resolver for SOAP references had been unreachable since it was written, and its unit test passed. `describe microflow` printed the right service and mapping names throughout", "cause": "`resolveWebServiceReference` looked the service up by ELEMENT ID among units of type `WebServices$ImportedWebService`. It could not match on two independent counts: the stored $Type is `WebServices$ImportedServiceImpl` (ImportedWebService is the SDK name, and nothing is stored under it), and the value compared was never an id \u2014 ImportedService is a BY_NAME_REFERENCE, so it already held `Clients.OrderSoapClient`. Same for the two mapping resolvers. Every call fell through to a fallback that returned the stored string, which is the correct answer", "file": "`mdl/executor/cmd_microflows_format_action.go` (formatWebServiceCallAction; the five resolvers removed)", "insight": "**A resolver whose fallback is the correct answer is indistinguishable in its output from one that works \u2014 so only the input side can prove it runs.** Nothing in the DESCRIBE text could ever have been wrong, which is why this survived: the test that covered it, `TestFormatAction_WebServiceCallResolvesKnownReferences`, built a world where ServiceID WAS a unit id and the unit type WAS `ImportedWebService`, neither of which occurs in any project \u2014 the green test asserted the fiction, not the code. The replacement inverts it: the mock backend calls `t.Fatal` if it is consulted at all, so the test fails unless no lookup happens. **The wider fact, measured, is that the structured DESCRIBE branch is unreachable for real SOAP calls anyway**: all three of ako/TestApp's carry 15 keys and `webServiceActionRequiresRawBSON` supports 9, and mxcli's own writer emits the same 15, so every SOAP call on either engine describes as `call web service raw ''`. A branch nothing reaches cannot be validated by any amount of passing tests over it. **Check a BY_NAME_REFERENCE before writing a resolver**: `modelsdk/gen/*/refs.go` states the kind (`codec.RefByName` here), and an existing test two packages away was already passing `Mod.Service` as the value", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "`send mapping X` on a SOAP call parsed, `exec` reported success, and the mapping name appeared ZERO times in the written document \u2014 on both engines. mxbuild then rejected the call as `[CE0369] \"Cannot use simple request body, as the operation's body is complex\"`. Separately, an operation taking parameters had no MDL syntax at all and built as `[CE0178] \"Body parameter mapping needs to be refreshed.\"`", "cause": "Both writers emitted an unconditional empty `Microflows$SimpleRequestHandling` for `RequestBodyHandling`. It is a POLYMORPHIC child holding EITHER the operation's arguments (SimpleRequestHandling + WebServiceOperationSimpleParameterMapping entries) or an export mapping (`Microflows$MappingRequestHandling`, NOT the `Mendix$AdvancedRequestHandling` legacy's comment named \u2014 that type is in none of the three ako/TestApp reference documents). The reader compounded it by looking for `RequestHandling`/`ExportMappingCall`, keys no real document carries; the stored key is `RequestBodyHandling`", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go`, `sdk/mpr/writer_microflow_actions.go` (webServiceRequestBody), `mdl/executor/webservice_names.go` (webServiceParameterPath), `mdl/grammar/domains/MDLMicroflow.g4`", "insight": "**Two gaps that look independent can be one polymorphic property, and finding that out is what makes the validation possible.** Arguments and the send mapping are the two branches of `RequestBodyHandling`, so the mutual-exclusion rule (MDL-SOAP01) only becomes enforceable once BOTH exist \u2014 implementing either alone means `check` can refuse a combination it cannot offer an alternative to. **The stored ParameterPath is derivable, which is what makes readable syntax possible**: `escape(operation.RequestBodyElementName) + \"|\" + name` reads the element off `Description.Services[].Operations[]` of the imported service, so MDL says `OrderId` and not `http%3A//www.example.com/:GetOrder|OrderId`. Escaping is per SEGMENT \u2014 `:` inside a segment becomes %3A, the separator `:` and the `/`es are left alone \u2014 and a segment containing `%` is REFUSED because Mendix's escaping of it is unmeasured. **The typed-array marker is the silent trap**: a populated ParameterMappings list leads with marker 2 and `codec.lookupListMarker` DEFAULTS TO 3, so without a `RegisterListMarker` the arguments serialize under the wrong array version \u2014 invisible to mxbuild, fatal to Studio Pro. Control: stubbing the registration makes the test report `marker = 3, want int32(2)`", "refs": []} +{"area": "mdl/executor", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec`, the documented copy operation, silently turned OFF a microflow's \"apply entity access\" (16/342 microflows across 4 projects, 11.14.0) and turned a blocking `show message` into a non-blocking one (16 microflows). Neither `mxcli check` nor mxbuild says anything: the model is valid either way, and the only consequence is that a constrained user behaves differently", "cause": "TWO causes, one symptom. ApplyEntityAccess was HARDCODED false in both writers and `microflows.Microflow` had no field, so the read side dropped it first \u2014 while `microflows.Rule` carried it correctly in the same codebase (and its CREATE path never set it, so rules had the same bug from the other end). ShowMessageAction.Blocking was carried perfectly by BOTH engines; MDL simply had no keyword, so DESCRIBE could not emit it and the re-parse set false", "file": "`sdk/microflows/microflows.go`, `mdl/backend/modelsdk/microflow.go`+`microflow_write.go`, `sdk/mpr/parser_microflow.go`+`writer_microflow.go`, `mdl/executor/apply_entity_access.go`, `mdl/executor/cmd_microflows_build.go`, `mdl/executor/cmd_rules_create.go`, `mdl/grammar/MDLLexer.g4` (BLOCKING)", "insight": "**Carrying a property through the writers is only half a round-trip fix when the middle is MDL text.** describe -> exec rebuilds from the STATEMENT, so a setting the statement never names is gone however well the storage layer handles it \u2014 which is why ApplyEntityAccess needed BOTH preserve-on-rewrite (for CREATE OR REPLACE) and an annotation (for the copy case, where there is nothing to preserve from). **Absent must not mean false for a security setting**: the AST field is a *bool so `@applyentityaccess(false)` and silence are distinguishable, the same rule @excluded (#914) and the doc comment (#1018) already follow. No grammar change was needed \u2014 `annotationValue` already accepts a literal, so a parameterised annotation is free. **Adding a lexer keyword needs a keyword-rule entry and a test**: BLOCKING would otherwise stop `blocking` being usable as a parameter name. **The fix is only believable with the audit re-run as its verification** \u2014 three unit-test controls plus the same 342-microflow measurement showing both properties stopped moving. That re-run also surfaced CE0709 \"Sequence flow is not accepted by origin or destination\" on a whole-project round trip, which the pre-fix binary reproduces identically: a separate, pre-existing flow-graph defect that would have been easy to misattribute to this change", "refs": []} diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 3615d291b..21675580d 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -160,6 +160,8 @@ begin end; ``` +The sibling document annotation is **`@applyentityaccess`** — runs the flow under the current user's entity access rules rather than with full access, with the same absent-preserves rule and an explicit `(false)` to turn it off ([pitfalls](reference/pitfalls.md#apply-entity-access)). + Two rules follow, and both are enforced rather than documented-and-hoped: - **An absent `@excluded` never un-excludes.** It means "the script does not say", @@ -666,6 +668,9 @@ $Name = $Other/Name on error { return 'lookup failed'; }; change $Order (Status = Shipped) on error { log error 'could not ship'; return; }; log info node 'App' 'starting' on error { return; }; show message 'saved' on error { return; }; + +-- BLOCKING halts the client until dismissed; after `objects`, before `on error`. +show message 'Hello {1}' type Warning objects [$Name] blocking; validation feedback $Order/Total message 'must be positive' on error { return; }; show page Module.Page on error { return; }; close page on error { return; }; diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index e7027e7e2..728e11dd9 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -507,3 +507,32 @@ end; | CE0008 | No action defined | Define action for activity | | CW0094 | Variable never used | Remove unused variables or use them | | MDL | Variable not declared | Use `declare $var type = value;` before SET | + +## Apply entity access + +`@applyentityaccess` before a `create microflow` (or `create rule`) sets Studio +Pro's **"Apply entity access"** checkbox: the flow runs under the **current +user's** entity access rules instead of with full access. + +```mdl +@applyentityaccess +create microflow MyModule.ReadOwnOrders () +returns list of MyModule.Order +begin + retrieve $Orders from MyModule.Order; + return $Orders; +end; +``` + +It is a **security** setting and it only ever narrows, so the rules mirror +`@excluded`: + +- **An absent `@applyentityaccess` never turns it off.** It means "the script does + not say", so a `create or modify` that omits it preserves whatever is stored. + Before this was carried, every rewrite cleared the flag — *widening* what the + microflow could read and write, with `mxcli check`, mxbuild and the model all + perfectly happy. Measured across 342 microflows in 4 projects: every microflow + storing the flag came back without it, and nothing anywhere reported it. +- **Turning it off is explicit**: `@applyentityaccess(false)`. +- **Not available on a nanoflow.** A nanoflow runs in the client and Mendix stores + no such property, so the annotation would parse and do nothing. diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index 227fd6752..635caa78f 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -173,6 +173,7 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "EMPTY", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "OBJECT", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "OBJECTS", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, + {Label: "BLOCKING", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, // Widget keyword {Label: "PAGES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index edef41907..1f2c85ee6 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -304,10 +304,17 @@ func init() { "@curve(from: (40, -90), to: (-40, 90)) -- the flow's bezier control vectors\n" + "@merge(x, y) -- the implicit merge that closes a split\n" + "@caption 'text'\n@color Green\n@annotation 'a note'\n@excluded\n" + + "@applyentityaccess | @applyentityaccess(false) -- DOCUMENT-level, before CREATE MICROFLOW/RULE\n" + "@annotation(id: n1, text: 'a note', position: (x, y), size: (w, h))\n" + "@annotation(id: n1) -- attaches THAT note to another activity\n\n" + "An unrecognised @name is an error (MDL059): it would parse and do nothing,\n" + "so a typo of @position would silently discard the layout.\n\n" + + "@excluded and @applyentityaccess are DOCUMENT annotations — they go before\n" + + "CREATE, not on a statement. @applyentityaccess runs the flow under the\n" + + "current user's entity access rules instead of with full access; it is a\n" + + "SECURITY setting and only ever narrows, so an ABSENT annotation PRESERVES\n" + + "whatever is stored and @applyentityaccess(false) is how a script turns it\n" + + "off. A nanoflow has no such property (it runs in the client).\n\n" + "Mendix stores no waypoints — a flow's shape is two control vectors, each a\n" + "pixel offset from its end of the line. (0, 0) at both ends is straight.\n" + "@position on a split belongs to the SPLIT, so its end-if join has its own\n" + diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 79eb76668..071016cfb 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -538,10 +538,12 @@ it is for pages. | Show page | `show page Module.PageName ($Param = $value);` | Also accepts `(Param: $value)` | | Close page | `close page;` | | | Download file | `download file $FileDocument [show in browser];` | Streams a `System.FileDocument` | +| Show message | `show message 'text' [type Information\|Warning\|Error] [objects [$a, $b]] [blocking];` | `blocking` halts the client until the user dismisses it — Studio Pro's checkbox. It goes after `objects` and before `on error`. Without it, a describe → exec round trip turned a blocking message into a non-blocking one (16 microflows measured) | | Database connection credentials | `connection string @Mod.Const`, `username @Mod.Const`, `password @Mod.Const` | Constant **references** only. A literal writes an unopenable project — MDL058 | | Synchronize (nanoflow only) | `synchronize all;` / `synchronize unsynchronized;` / `synchronize $Obj, $List;` | Offline sync. `unsynchronized` needs Mendix 9.4+. In a microflow this is MDL057 / CE0009 | | Validation | `validation feedback $entity/attribute message 'message';` | Requires attribute path + MESSAGE | | Log | `log info\|warning\|error [node 'name'] 'message';` | | +| Apply entity access | `@applyentityaccess` / `@applyentityaccess(false)` before `create microflow` or `create rule` | Runs the flow under the **current user's** entity access rules instead of with full access. A **security** setting and only ever narrowing, so an ABSENT annotation **preserves** what is stored rather than clearing it — the same rule as `@excluded`. Not available on a nanoflow: it runs in the client and Mendix stores no such property | | Position | `@position(x, y)` | Canvas position (before activity) | | Parameter position | `@position(x, y)` before a parameter, **inside** the `( … )` list | The only annotation a parameter takes. Omit it and parameters form a row at 200;53, 300;53, …; a parameter off that row is treated as hand-placed, survives a rewrite, and is emitted by DESCRIBE (#993) | | Start event | `@start(x, y)` | Canvas position of the start, on the **first** statement. Omit it and the start is placed one spacing unit left of the first activity and MOVES with it on a rewrite; a start that is not at that derived spot is treated as hand-placed, survives a rewrite, and is emitted by DESCRIBE (#951) | diff --git a/docs/12-bug-reports/microflow-rewrite-property-audit.md b/docs/12-bug-reports/microflow-rewrite-property-audit.md index fe1496bfb..e46aa89a7 100644 --- a/docs/12-bug-reports/microflow-rewrite-property-audit.md +++ b/docs/12-bug-reports/microflow-rewrite-property-audit.md @@ -1,5 +1,10 @@ # Bug Report: what a microflow rewrite loses — a property-by-property audit +> **Status: both findings fixed.** Re-running the audit across the same 342 +> microflows shows `ApplyEntityAccess` and `ShowMessageAction.Blocking` no longer +> move. See [Fix](#fix) for what changed and how it was verified. The audit body +> below is left as written so the measurement it describes stays readable. + ## Summary `describe microflow` → `exec` is the documented copy operation, and it does not @@ -77,6 +82,66 @@ show message 'The password has been updated.' type Information; and the re-parse sets `Blocking: false`. Anything rewriting the microflow from **stored BSON** keeps it; only the round trip through MDL text loses it. +## Fix + +Both were fixed in one change; they needed different repairs because they had +different causes. + +**`ApplyEntityAccess`** — the property now exists on `microflows.Microflow`, is +read and written by both engines, and the executor **preserves** a stored value +on a rewrite that does not mention it. That last part is the load-bearing half: +`describe` → `exec` never states the setting, so carrying it through the writers +alone would not have helped. + +The documented **copy** operation (describe → rename → exec) has nothing to +preserve from, so MDL also gained a way to say it. `@applyentityaccess` before +`create microflow` / `create rule`, with `@applyentityaccess(false)` to clear it +— the same absent-preserves rule as `@excluded` (#914) and the doc comment +(#1018), and no grammar change, since `annotationValue` already accepts a +literal. A nanoflow is deliberately excluded: it runs in the client and Mendix +stores no such property, so the annotation would parse and do nothing. + +Rules had the same gap from the other end — `rule_write.go` plumbed the property +through while nothing ever set it — and are fixed alongside. + +**`ShowMessageAction.Blocking`** — a `blocking` modifier on `show message`, after +the `objects` clause and before `on error`. The model already carried it on both +engines, so only the grammar, visitor, builder and describe formatter were +missing. `BLOCKING` is listed in the `keyword` rule, so `blocking` remains usable +as an ordinary identifier — pinned by a test. + +### Verification + +- **The audit itself, re-run**: `ApplyEntityAccess` and `Blocking` are gone from + the changed list across all four projects, 342 microflows. +- **The three cases each work**: a rewrite with no annotation preserves a stored + `true`; a copy carrying `@applyentityaccess` creates one with `true`; + `@applyentityaccess(false)` creates one with `false`. +- **Controls**: reverting each of the three code changes in turn makes its test + fail with the reported symptom — `ApplyEntityAccess lost on round-trip`, + `written ApplyEntityAccess = false, want true`, and a describe emitting + `show message 'Saved.' type Information;` without the modifier. + +### Still open, and newly measured + +Round-tripping **every** microflow in TestApp produces a project that does not +build: **CE0709** "Sequence flow is not accepted by origin or destination". The +control is the same operation on the pre-fix binary, which gives the identical +error — so this is pre-existing flow-graph drift, not a consequence of these +fixes, and it belongs to the "flow graph changed in 40/42" row rather than to +either property above. It is a stronger statement of the reviewability problem +than the 417-line diff, and wants its own investigation. + +`ConcurrenyErrorMessage` and `ConcurrencyErrorMicroflow` are still hardcoded. +Neither is a demonstrated loss (see the benign table), so closing those holes +needs a reference project that actually sets them. + +One gap this change does not close: a **typo'd document annotation** parses and +does nothing. MDL059 covers statement annotations only, so `@applyentityacces` +is silent — as `@excluded` already was. Both typos fail safe here (an unset flag +on a create means off; on a rewrite it means preserve), which is why this is +noted rather than fixed. + ## Method, and what it is worth For every microflow in each project: dump the stored BSON, `describe` it, `exec` @@ -119,6 +184,8 @@ worth a separate issue. ## Suggested fixes, in severity order +*(1 and 2 are done — see [Fix](#fix). Left in place as the reasoning behind what was built.)* + 1. **`ApplyEntityAccess`** — add the field to `microflows.Microflow`, read it on both engines, and give MDL a way to say it. The rule path is the precedent to copy. Until the syntax exists, the safe interim is **guard-don't-drop** diff --git a/mdl-examples/doctype-tests/02-microflow-examples.mdl b/mdl-examples/doctype-tests/02-microflow-examples.mdl index 18735093f..6801c325f 100644 --- a/mdl-examples/doctype-tests/02-microflow-examples.mdl +++ b/mdl-examples/doctype-tests/02-microflow-examples.mdl @@ -2564,6 +2564,28 @@ begin end; / +/** + * M082b: BLOCKING messages, and APPLY ENTITY ACCESS on the microflow itself. + * + * `blocking` is Studio Pro's checkbox on a message action — the client halts + * until the user dismisses it. It goes after the OBJECTS clause and before + * ON ERROR, so it combines with both. (Continue error handling is a separate + * matter: Mendix rejects it on a message action, MDL076 / CE6035.) + * + * `@applyentityaccess` makes the microflow run under the CURRENT USER's entity + * access rules instead of with full access. It is a security setting and only + * ever narrows, so an ABSENT annotation preserves whatever is stored rather + * than clearing it; `@applyentityaccess(false)` is how a script turns it off. + */ +@applyentityaccess +create microflow MfTest.M082b_BlockingMessages ($Name : string) +begin + show message 'Saved.' type Information blocking; + show message 'Hello {1}' type Warning objects [$Name] blocking; + return; +end; +/ + /** * M083: Explicit canvas positions on activities. * @position(x, y) controls where the activity appears on the canvas. diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index ef7b7ed51..c72e710b2 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -65,6 +65,15 @@ type CreateMicroflowStmt struct { Folder string // Folder path within module (e.g., "Resources/Images") CreateOrModify bool Excluded bool // @excluded — document excluded from project + // ApplyEntityAccess is Studio Pro's "apply entity access" checkbox, set by + // `@applyentityaccess` / `@applyentityaccess(false)`. + // + // A POINTER because absent and false are different answers: an absent + // annotation preserves what is stored (the setting is model state, like + // @excluded), while an explicit false clears it. A plain bool would make + // every rewrite that did not mention it turn the setting off, which is the + // bug this field exists to fix. + ApplyEntityAccess *bool // Expose holds the EXPOSED AS … ACTION clauses. A microflow has two toolbox // entries — one for the microflow editor, one for the workflow editor — so // there can be one of each. @@ -119,6 +128,11 @@ type CreateNanoflowStmt struct { Folder string // Folder path within module CreateOrModify bool Excluded bool // @excluded — document excluded from project + // No ApplyEntityAccess: a nanoflow runs in the CLIENT and Mendix stores no + // such property on Nanoflows$Nanoflow (modelsdk/gen declares the accessor + // on Microflow and Rule only). Carrying it here would give the annotation + // somewhere to parse and nothing to do. + // // Expose is parsed but refused: only Microflows$Microflow carries the toolbox // properties. Accepting it in the grammar and explaining the refusal beats a // parse error that says only "no viable alternative". @@ -142,6 +156,15 @@ type CreateRuleStmt struct { Folder string // Folder path within module CreateOrModify bool Excluded bool // @excluded — document excluded from project + // ApplyEntityAccess is Studio Pro's "apply entity access" checkbox, set by + // `@applyentityaccess` / `@applyentityaccess(false)`. + // + // A POINTER because absent and false are different answers: an absent + // annotation preserves what is stored (the setting is model state, like + // @excluded), while an explicit false clears it. A plain bool would make + // every rewrite that did not mention it turn the setting off, which is the + // bug this field exists to fix. + ApplyEntityAccess *bool // Expose is parsed but refused — see CreateNanoflowStmt.Expose. Expose []ExposeActionClause } @@ -927,6 +950,7 @@ type ShowMessageStmt struct { Message Expression // The message text (string template) Type string // Information, Warning, Error (default: Information) TemplateArgs []Expression // Template arguments for message placeholders {1}, {2}, etc. + Blocking bool // BLOCKING — the message halts the client until dismissed Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause } diff --git a/mdl/backend/modelsdk/microflow.go b/mdl/backend/modelsdk/microflow.go index f18d93ebf..f3fe7fc0a 100644 --- a/mdl/backend/modelsdk/microflow.go +++ b/mdl/backend/modelsdk/microflow.go @@ -200,6 +200,11 @@ func microflowFromGen(mf *genMf.Microflow, containerID model.ID) *microflows.Mic // message configured → mx check CE4899. AllowConcurrentExecution: mf.AllowConcurrentExecution(), MarkAsUsed: mf.MarkAsUsed(), + // The same class again, and this one is a security setting: without it + // a rewrite turned "apply entity access" OFF, widening what the + // microflow may read and write. mx check and mxbuild are both silent, + // because the model is valid either way. + ApplyEntityAccess: mf.ApplyEntityAccess(), } out.ID = model.ID(mf.ID()) // AllowedModuleRoles (BY_NAME role references) — without these DESCRIBE omits diff --git a/mdl/backend/modelsdk/microflow_roundtrip_flags_test.go b/mdl/backend/modelsdk/microflow_roundtrip_flags_test.go index 2efad9cdb..fc645ced3 100644 --- a/mdl/backend/modelsdk/microflow_roundtrip_flags_test.go +++ b/mdl/backend/modelsdk/microflow_roundtrip_flags_test.go @@ -56,3 +56,34 @@ func TestMicroflowRoundTrip_ConcurrentExecutionFlags(t *testing.T) { t.Error("MarkAsUsed lost on round-trip (want true)") } } + +// TestMicroflowRoundTrip_ApplyEntityAccess is the third property in this struct +// to go the way #723 §A describes, and the only one with a security consequence. +// +// "Apply entity access" makes a microflow run under the current user's entity +// access rules rather than with full access, so it only ever NARROWS. Both +// writers hardcoded false and microflowFromGen did not read it back, so every +// rewrite turned it off — widening what the microflow may read and write, with +// nothing to report it: `mxcli check` is quiet, mxbuild is quiet, and the model +// is valid either way. +// +// Measured across 342 microflows in 4 projects (11.14.0): every microflow +// storing true came back false. The write half is the assertion below; the +// executor's preserve-on-rewrite rule is TestCarriedApplyEntityAccess. +func TestMicroflowRoundTrip_ApplyEntityAccess(t *testing.T) { + mf := µflows.Microflow{Name: "ACT_Secured", ApplyEntityAccess: true} + mf.ID = model.ID("mf-2") + + if got := roundTripMicroflow(t, mf); !got.ApplyEntityAccess { + t.Error("ApplyEntityAccess lost on round-trip — the microflow now runs with " + + "FULL access instead of the user's (want true)") + } + + // The other direction has to survive too: a stored false must not become + // true, or the fix would be a different silent change in the same place. + off := µflows.Microflow{Name: "ACT_Plain"} + off.ID = model.ID("mf-3") + if got := roundTripMicroflow(t, off); got.ApplyEntityAccess { + t.Error("ApplyEntityAccess invented on round-trip (want false)") + } +} diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index de10eb78c..47b61ede7 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -205,7 +205,9 @@ func microflowToGen(mf *microflows.Microflow, major int) *genMf.Microflow { out.SetExcluded(mf.Excluded) out.SetExportLevel("Hidden") out.SetAllowConcurrentExecution(mf.AllowConcurrentExecution) - out.SetApplyEntityAccess(false) + // Carried, not hardcoded. This was `false` unconditionally, which silently + // turned a microflow's "apply entity access" OFF on every rewrite. + out.SetApplyEntityAccess(mf.ApplyEntityAccess) out.SetMarkAsUsed(mf.MarkAsUsed) out.SetConcurrencyErrorMicroflowQualifiedName("") out.SetConcurrencyErrorMessage(genTexts.NewText()) // empty Texts$Text (Items=[3] via default) diff --git a/mdl/executor/apply_entity_access.go b/mdl/executor/apply_entity_access.go new file mode 100644 index 000000000..92ac2852e --- /dev/null +++ b/mdl/executor/apply_entity_access.go @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +// carriedApplyEntityAccess decides a rewrite's "apply entity access" setting. +// +// An ABSENT annotation preserves what is stored; `@applyentityaccess` and +// `@applyentityaccess(false)` set it. That asymmetry is deliberate and it is the +// same rule `@excluded` and the doc comment follow (#914, #1018): the setting is +// model state, not script state, so a statement that does not mention it must +// not decide it. +// +// It matters more here than for those two because this is a SECURITY setting and +// it only ever narrows. Defaulting an unstated rewrite to false turned "apply +// entity access" OFF — widening what the microflow may read and write — and +// nothing reported it: `mxcli check` is quiet, mxbuild is quiet, and the model +// is valid either way. Measured across 342 microflows in 4 projects: every +// microflow storing true came back false. +// +// On a CREATE there is nothing stored, so stored is false and a fresh microflow +// gets Studio Pro's default (326 of 342 in that corpus are false). +func carriedApplyEntityAccess(stated *bool, stored bool) bool { + if stated != nil { + return *stated + } + return stored +} diff --git a/mdl/executor/apply_entity_access_test.go b/mdl/executor/apply_entity_access_test.go new file mode 100644 index 000000000..66ebecb09 --- /dev/null +++ b/mdl/executor/apply_entity_access_test.go @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import "testing" + +// TestCarriedApplyEntityAccess pins the three-way rule, and the middle row is +// the bug: an unstated rewrite must PRESERVE, not default to false. +// +// "Apply entity access" makes a microflow run under the current user's access +// rules, so clearing it WIDENS what the microflow may read and write — with +// `mxcli check` quiet, mxbuild quiet, and the model valid either way. Measured +// across 342 microflows in 4 projects: every microflow storing true came back +// false, because both writers hardcoded it. +func TestCarriedApplyEntityAccess(t *testing.T) { + yes, no := true, false + for _, tc := range []struct { + name string + stated *bool + stored bool + want bool + }{ + {"unstated rewrite preserves a stored true", nil, true, true}, + {"unstated rewrite preserves a stored false", nil, false, false}, + {"@applyentityaccess sets it", &yes, false, true}, + {"@applyentityaccess(false) clears it", &no, true, false}, + // A CREATE has nothing stored, so a fresh microflow gets Studio Pro's + // default (326 of 342 in the corpus are false). + {"fresh microflow defaults off", nil, false, false}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := carriedApplyEntityAccess(tc.stated, tc.stored); got != tc.want { + t.Errorf("carriedApplyEntityAccess(%v, %v) = %v, want %v", + tc.stated, tc.stored, got, tc.want) + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_build.go b/mdl/executor/cmd_microflows_build.go index 74bd19b1a..db5976396 100644 --- a/mdl/executor/cmd_microflows_build.go +++ b/mdl/executor/cmd_microflows_build.go @@ -120,6 +120,10 @@ func buildMicroflowFromStmt(ctx *ExecContext, s *ast.CreateMicroflowStmt, opts b // Excluded is model state, not script state: an absent @excluded must not // clear a stored exclusion (#914). existingExcluded := false + // Same reasoning for "apply entity access", and it matters more: it is a + // SECURITY setting, so an absent annotation must preserve a stored true + // rather than widening what the microflow may read and write. + existingApplyEntityAccess := false var existingDocumentation string preserveDocumentation := false var existingActionInfo, existingWorkflowInfo *types.MicroflowActionInfo @@ -144,6 +148,7 @@ func buildMicroflowFromStmt(ctx *ExecContext, s *ast.CreateMicroflowStmt, opts b existingAllowedRoles = cloneRoleIDs(existing.AllowedModuleRoles) preserveAllowedRoles = true existingExcluded = existing.Excluded + existingApplyEntityAccess = existing.ApplyEntityAccess // The toolbox entries hold four PNG bitmaps MDL cannot name, so a // rewrite carries them rather than rebuilding from the clause. existingActionInfo = existing.MicroflowActionInfo @@ -204,6 +209,7 @@ func buildMicroflowFromStmt(ctx *ExecContext, s *ast.CreateMicroflowStmt, opts b AllowConcurrentExecution: true, // Default: allow concurrent execution MarkAsUsed: false, Excluded: s.Excluded || existingExcluded, + ApplyEntityAccess: carriedApplyEntityAccess(s.ApplyEntityAccess, existingApplyEntityAccess), } if preserveDocumentation { mf.Documentation = carriedDocumentation(s.DocumentationSet, s.Documentation, existingDocumentation) diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index d5a437188..5d0a76e22 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -1125,6 +1125,7 @@ func (fb *flowBuilder) addShowMessageAction(s *ast.ShowMessageStmt) model.ID { Template: template, Type: msgType, TemplateParameters: templateParams, + Blocking: s.Blocking, } activity := µflows.ActionActivity{ diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index 40382dcad..e5084465d 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -789,6 +789,14 @@ func formatAction( if len(a.TemplateParameters) > 0 { result += " objects [" + strings.Join(a.TemplateParameters, ", ") + "]" } + // Without this, a describe -> exec round trip turned a BLOCKING message + // box into a non-blocking one. The model carried Blocking on both + // engines all along; MDL simply had no word for it, which is why the + // loss happened in the middle of a path where every other layer was + // correct. + if a.Blocking { + result += " blocking" + } return result + ";" case *microflows.DownloadFileAction: diff --git a/mdl/executor/cmd_microflows_format_action_test.go b/mdl/executor/cmd_microflows_format_action_test.go index b7d229fa1..427f7337b 100644 --- a/mdl/executor/cmd_microflows_format_action_test.go +++ b/mdl/executor/cmd_microflows_format_action_test.go @@ -1424,3 +1424,25 @@ func TestFormatAction_WebServiceCallArgumentWithoutAName(t *testing.T) { t.Errorf("emitted a partial argument list: %q", got) } } + +// TestFormatAction_ShowMessageBlocking — DESCRIBE has to emit `blocking`, or the +// round trip turns a blocking message box into a non-blocking one. +// +// The model carried Blocking on both engines all along; the loss was here, in +// the one layer that had no word for it. +func TestFormatAction_ShowMessageBlocking(t *testing.T) { + e := newTestExecutor() + msg := func(blocking bool) *microflows.ShowMessageAction { + return µflows.ShowMessageAction{ + Type: microflows.MessageTypeInformation, + Blocking: blocking, + Template: &model.Text{Translations: map[string]string{"en_US": "Saved."}}, + } + } + if got := e.formatAction(msg(true), nil, nil); got != "show message 'Saved.' type Information blocking;" { + t.Errorf("blocking message = %q", got) + } + if got := e.formatAction(msg(false), nil, nil); got != "show message 'Saved.' type Information;" { + t.Errorf("non-blocking message = %q", got) + } +} diff --git a/mdl/executor/cmd_microflows_show.go b/mdl/executor/cmd_microflows_show.go index 395a98b38..cdc95bd7d 100644 --- a/mdl/executor/cmd_microflows_show.go +++ b/mdl/executor/cmd_microflows_show.go @@ -246,6 +246,13 @@ func describeMicroflow(ctx *ExecContext, name ast.QualifiedName) error { if targetMf.Excluded { lines = append(lines, "@excluded") } + // A SECURITY setting, and only ever narrowing: DESCRIBE has to emit it or + // a describe -> rename -> exec copy silently runs with full access. An + // absent annotation preserves the stored value on a REWRITE, but a copy + // has nothing to preserve from. + if targetMf.ApplyEntityAccess { + lines = append(lines, "@applyentityaccess") + } // CREATE MICROFLOW header qualifiedName := name.Module + "." + name.Name @@ -588,6 +595,13 @@ func renderMicroflowMDL( if mf.Excluded { lines = append(lines, "@excluded") } + // A SECURITY setting, and only ever narrowing: DESCRIBE has to emit it or + // a describe -> rename -> exec copy silently runs with full access. An + // absent annotation preserves the stored value on a REWRITE, but a copy + // has nothing to preserve from. + if mf.ApplyEntityAccess && flowType == "microflow" { + lines = append(lines, "@applyentityaccess") + } qualifiedName := name.Module + "." + name.Name if len(mf.Parameters) > 0 { @@ -1510,6 +1524,13 @@ func describeRule(ctx *ExecContext, name ast.QualifiedName) error { if target.Excluded { lines = append(lines, "@excluded") } + // A SECURITY setting, and only ever narrowing: DESCRIBE has to emit it or + // a describe -> rename -> exec copy silently runs with full access. An + // absent annotation preserves the stored value on a REWRITE, but a copy + // has nothing to preserve from. + if target.ApplyEntityAccess { + lines = append(lines, "@applyentityaccess") + } qualifiedName := name.Module + "." + name.Name if len(target.Parameters) > 0 { diff --git a/mdl/executor/cmd_rules_create.go b/mdl/executor/cmd_rules_create.go index 97d9272b0..bc37c9044 100644 --- a/mdl/executor/cmd_rules_create.go +++ b/mdl/executor/cmd_rules_create.go @@ -65,6 +65,10 @@ func execCreateRule(ctx *ExecContext, s *ast.CreateRuleStmt) error { // Excluded is model state, not script state: an absent @excluded must not // clear a stored exclusion (#914). existingExcluded := false + // Same as the microflow path: "apply entity access" is a SECURITY setting + // and model state, so an absent annotation preserves the stored value + // rather than widening what the rule may read. + existingApplyEntityAccess := false // Studio Pro's rule editor writes "Variable" here on both reference rules, so // a rule mxcli creates matches rather than storing an empty name that Studio // Pro would fill in on first edit. @@ -87,6 +91,7 @@ func execCreateRule(ctx *ExecContext, s *ast.CreateRuleStmt) error { existingID = existing.ID existingContainerID = existing.ContainerID existingExcluded = existing.Excluded + existingApplyEntityAccess = existing.ApplyEntityAccess existingDocumentation = existing.Documentation haveExisting = true // MDL has no surface for ReturnVariableName, and Studio Pro writes one @@ -115,6 +120,7 @@ func execCreateRule(ctx *ExecContext, s *ast.CreateRuleStmt) error { Documentation: s.Documentation, MarkAsUsed: false, Excluded: s.Excluded || existingExcluded, + ApplyEntityAccess: carriedApplyEntityAccess(s.ApplyEntityAccess, existingApplyEntityAccess), ReturnVariableName: existingReturnVariableName, } diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index 93a8c2b54..56d0e963b 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -251,6 +251,10 @@ WITH: W I T H; EMPTY: E M P T Y; OBJECT: O B J E C T; OBJECTS: O B J E C T S; +// SHOW MESSAGE … BLOCKING — Studio Pro's "blocking" checkbox on a message +// action. Listed in the `keyword` rule too, so `blocking` stays usable as an +// ordinary identifier. +BLOCKING: B L O C K I N G; // ============================================================================= // PAGE / WIDGET KEYWORDS diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index c0830f755..6bc6725f3 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -621,7 +621,7 @@ showHomePageStatement // SHOW MESSAGE 'Hello {1}' TYPE Information OBJECTS [$Name]; showMessageStatement - : SHOW MESSAGE expression (TYPE identifierOrKeyword)? (OBJECTS LBRACKET expressionList RBRACKET)? onErrorClause? + : SHOW MESSAGE expression (TYPE identifierOrKeyword)? (OBJECTS LBRACKET expressionList RBRACKET)? BLOCKING? onErrorClause? ; // SYNCHRONIZE ALL; diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index 70afbc53e..da0c6afd1 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -713,7 +713,7 @@ keyword | AFTER | BEFORE | DEFINE | FRAGMENT | FRAGMENTS | SLOT // General-purpose words (only tokens not already listed above) - | ACTION | BOTH | CONTEXT | DATA | FORMAT | ITEM | LIST + | ACTION | BLOCKING | BOTH | CONTEXT | DATA | FORMAT | ITEM | LIST | DEFINITION | IGNORE | MESSAGE | MOD | DIV | MULTIPLE | NONE | OBJECT | OBJECTS | OVERRIDABLE | ROOT | SINGLE | SQL | TEMPLATE | TEXT | TYPE | VALUE diff --git a/mdl/visitor/visitor_apply_entity_access_test.go b/mdl/visitor/visitor_apply_entity_access_test.go new file mode 100644 index 000000000..6b00d8f7b --- /dev/null +++ b/mdl/visitor/visitor_apply_entity_access_test.go @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func microflowStmt(t *testing.T, src string) *ast.CreateMicroflowStmt { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + for _, e := range errs { + t.Errorf("parse: %v", e) + } + t.FailNow() + } + for _, s := range prog.Statements { + if mf, ok := s.(*ast.CreateMicroflowStmt); ok { + return mf + } + } + t.Fatalf("no CreateMicroflowStmt in:\n%s", src) + return nil +} + +// TestApplyEntityAccessAnnotation — absent is NOT false. +// +// The field is a pointer precisely so the executor can tell "the script did not +// mention it" (preserve the stored setting) from "the script said off". A plain +// bool made every rewrite that omitted the annotation turn the security setting +// off, which is the bug this parses for. +func TestApplyEntityAccessAnnotation(t *testing.T) { + body := "create or modify microflow M.Secured ()\nbegin\n return;\nend;" + + if got := microflowStmt(t, body).ApplyEntityAccess; got != nil { + t.Errorf("absent annotation = %v, want nil (preserve the stored value)", *got) + } + + on := microflowStmt(t, "@applyentityaccess\n"+body).ApplyEntityAccess + if on == nil || !*on { + t.Errorf("@applyentityaccess = %v, want true", on) + } + + off := microflowStmt(t, "@applyentityaccess(false)\n"+body).ApplyEntityAccess + if off == nil || *off { + t.Errorf("@applyentityaccess(false) = %v, want false", off) + } + + // The explicit-true spelling is accepted too, so a script can be emphatic. + explicit := microflowStmt(t, "@applyentityaccess(true)\n"+body).ApplyEntityAccess + if explicit == nil || !*explicit { + t.Errorf("@applyentityaccess(true) = %v, want true", explicit) + } +} + +// TestApplyEntityAccessAnnotationOnRule — a rule carries the same property, and +// its create path had the same gap: rule_write.go plumbed it through while +// nothing ever set it. +func TestApplyEntityAccessAnnotationOnRule(t *testing.T) { + src := "@applyentityaccess\ncreate or modify rule M.IsOk ()\nreturns Boolean\nbegin\n return true;\nend;" + prog, errs := Build(src) + if len(errs) > 0 { + for _, e := range errs { + t.Errorf("parse: %v", e) + } + t.FailNow() + } + for _, s := range prog.Statements { + if r, ok := s.(*ast.CreateRuleStmt); ok { + if r.ApplyEntityAccess == nil || !*r.ApplyEntityAccess { + t.Errorf("rule @applyentityaccess = %v, want true", r.ApplyEntityAccess) + } + return + } + } + t.Fatal("no CreateRuleStmt parsed") +} diff --git a/mdl/visitor/visitor_microflow.go b/mdl/visitor/visitor_microflow.go index f60e52b78..fa3089038 100644 --- a/mdl/visitor/visitor_microflow.go +++ b/mdl/visitor/visitor_microflow.go @@ -56,6 +56,7 @@ func (b *Builder) ExitCreateMicroflowStatement(ctx *parser.CreateMicroflowStatem stmt.Excluded = true } } + stmt.ApplyEntityAccess = applyEntityAccessAnnotation(createStmt) } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) @@ -108,6 +109,9 @@ func (b *Builder) ExitCreateNanoflowStatement(ctx *parser.CreateNanoflowStatemen stmt.Excluded = true } } + // No @applyentityaccess on a nanoflow: it runs in the client and + // Nanoflows$Nanoflow stores no such property, so the annotation would + // parse and do nothing. } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) @@ -153,6 +157,7 @@ func (b *Builder) ExitCreateRuleStatement(ctx *parser.CreateRuleStatementContext stmt.Excluded = true } } + stmt.ApplyEntityAccess = applyEntityAccessAnnotation(createStmt) } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) @@ -411,3 +416,33 @@ func buildExposeBitmaps(clauses []parser.IExposeBitmapClauseContext) []ast.Expos } return out } + +// applyEntityAccessAnnotation reads `@applyentityaccess` off a document's +// annotations, or nil when it is absent. +// +// nil is not false: an absent annotation PRESERVES the stored setting, the way +// @excluded and the doc comment do, because the setting is model state rather +// than script state. Collapsing the two is the bug this exists to fix — a +// rewrite that did not mention it turned "apply entity access" off. +// +// The bare form means true; `@applyentityaccess(false)` clears it. No grammar +// change is needed for either: annotationValue already accepts a literal. +func applyEntityAccessAnnotation(createStmt parser.ICreateStatementContext) *bool { + if createStmt == nil { + return nil + } + for _, ann := range createStmt.AllAnnotation() { + annCtx := ann.(*parser.AnnotationContext) + if !strings.EqualFold(annCtx.AnnotationName().GetText(), "applyentityaccess") { + continue + } + value := true + if params := annCtx.AnnotationParams(); params != nil { + if strings.EqualFold(strings.TrimSpace(params.GetText()), "false") { + value = false + } + } + return &value + } + return nil +} diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index 9d12439bf..d8b781362 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -1278,6 +1278,12 @@ func buildShowMessageStatement(ctx parser.IShowMessageStatementContext) *ast.Sho } } + // BLOCKING — Studio Pro's "blocking" checkbox. The model carried this on + // both engines already; MDL had no word for it, so DESCRIBE could not emit + // it and a describe -> exec round trip turned a blocking message box into a + // non-blocking one (16 microflows measured across 4 projects). + stmt.Blocking = smCtx.BLOCKING() != nil + // Check for ON ERROR clause if errClause := smCtx.OnErrorClause(); errClause != nil { stmt.ErrorHandling = buildOnErrorClause(errClause) diff --git a/mdl/visitor/visitor_microflow_actions_blocking_test.go b/mdl/visitor/visitor_microflow_actions_blocking_test.go new file mode 100644 index 000000000..dcd3739a3 --- /dev/null +++ b/mdl/visitor/visitor_microflow_actions_blocking_test.go @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// TestShowMessageBlocking — `blocking` is Studio Pro's checkbox on a message +// action, and MDL had no word for it. +// +// The model carried Blocking on BOTH engines all along, so nothing in the +// storage layer was wrong; DESCRIBE simply could not emit it, and the re-parse +// set false. That is why the loss survived: every layer except the text was +// correct. Measured on 16 microflows across 4 projects. +func TestShowMessageBlocking(t *testing.T) { + blocking := firstStatement(t, "show message 'Saved.' type Information blocking;").(*ast.ShowMessageStmt) + if !blocking.Blocking { + t.Error("`blocking` not parsed") + } + + plain := firstStatement(t, "show message 'Saved.' type Information;").(*ast.ShowMessageStmt) + if plain.Blocking { + t.Error("Blocking set on a statement that did not say it") + } + + // It sits after the OBJECTS clause and before ON ERROR, so all three + // combine. (Rollback, not Continue: Mendix rejects Continue error handling + // on a message action, which MDL076 refuses before the write.) + full := firstStatement(t, + "show message 'Hi {1}' type Warning objects [$Name] blocking on error rollback;").(*ast.ShowMessageStmt) + if !full.Blocking || len(full.TemplateArgs) != 1 || full.ErrorHandling == nil { + t.Errorf("combined clauses = %+v", full) + } +} + +// TestBlockingIsStillUsableAsAnIdentifier — adding a lexer token risks turning +// an ordinary name into a keyword. BLOCKING is listed in the `keyword` rule, so +// a parameter may still be called "blocking". +func TestBlockingIsStillUsableAsAnIdentifier(t *testing.T) { + prog, errs := Build("create microflow M.F (blocking: Boolean)\nbegin\n return;\nend;") + if len(errs) > 0 { + t.Fatalf("a parameter named `blocking` no longer parses: %v", errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + if len(mf.Parameters) != 1 || mf.Parameters[0].Name != "blocking" { + t.Errorf("parameters = %+v", mf.Parameters) + } +} diff --git a/sdk/microflows/microflows.go b/sdk/microflows/microflows.go index 562f83a2f..6f87f7fd8 100644 --- a/sdk/microflows/microflows.go +++ b/sdk/microflows/microflows.go @@ -17,6 +17,18 @@ type Microflow struct { AllowConcurrentExecution bool `json:"allowConcurrentExecution"` MarkAsUsed bool `json:"markAsUsed"` Excluded bool `json:"excluded"` + // ApplyEntityAccess makes the microflow run under the current user's entity + // access rules instead of with full access — Studio Pro's "Apply entity + // access" checkbox. + // + // It is a SECURITY setting and it is only ever narrowing, so losing it + // widens what the microflow may read and write with nothing to show for it: + // the model stays valid, the app builds, and only a constrained user + // behaves differently. Both writers used to hardcode false and this struct + // had no field at all, so every rewrite cleared it — the third property in + // this struct to go that way, after AllowConcurrentExecution and + // MarkAsUsed (#723 §A). + ApplyEntityAccess bool `json:"applyEntityAccess"` // Return type ReturnType DataType `json:"returnType,omitempty"` diff --git a/sdk/mpr/parser_microflow.go b/sdk/mpr/parser_microflow.go index 9f4fc6753..070b572fe 100644 --- a/sdk/mpr/parser_microflow.go +++ b/sdk/mpr/parser_microflow.go @@ -56,6 +56,12 @@ func ParseMicroflowFromRaw(raw map[string]any, unitID, containerID model.ID) *mi if excluded, ok := raw["Excluded"].(bool); ok { mf.Excluded = excluded } + // A security setting: without reading it, a rewrite turned "apply entity + // access" OFF, widening what the microflow may read and write with nothing + // reporting it. + if applyEntityAccess, ok := raw["ApplyEntityAccess"].(bool); ok { + mf.ApplyEntityAccess = applyEntityAccess + } // Parse allowed module roles (BY_NAME references) allowedRoles := extractBsonArray(raw["AllowedModuleRoles"]) diff --git a/sdk/mpr/writer_microflow.go b/sdk/mpr/writer_microflow.go index 821526398..55bdd607a 100644 --- a/sdk/mpr/writer_microflow.go +++ b/sdk/mpr/writer_microflow.go @@ -93,7 +93,10 @@ func (w *Writer) serializeMicroflow(mf *microflows.Microflow) ([]byte, error) { {Key: "$Type", Value: "Microflows$Microflow"}, {Key: "AllowConcurrentExecution", Value: mf.AllowConcurrentExecution}, {Key: "AllowedModuleRoles", Value: allowedModuleRolesArray(mf.AllowedModuleRoles)}, - {Key: "ApplyEntityAccess", Value: false}, + // Carried, not hardcoded — see the modelsdk twin. A hardcoded false + // turned "apply entity access" OFF on every rewrite, widening what the + // microflow may read and write. + {Key: "ApplyEntityAccess", Value: mf.ApplyEntityAccess}, {Key: "ConcurrencyErrorMicroflow", Value: ""}, {Key: "ConcurrenyErrorMessage", Value: bson.D{ {Key: "$ID", Value: idToBsonBinary(generateUUID())}, diff --git a/sdk/mpr/writer_microflow_flags_test.go b/sdk/mpr/writer_microflow_flags_test.go new file mode 100644 index 000000000..8d449b18c --- /dev/null +++ b/sdk/mpr/writer_microflow_flags_test.go @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" + "go.mongodb.org/mongo-driver/bson" +) + +// TestMicroflowApplyEntityAccessRoundTrip is the legacy half of the security +// fix, and exists to keep the two engines from drifting — the modelsdk twin is +// TestMicroflowRoundTrip_ApplyEntityAccess. +// +// This writer wrote `{Key: "ApplyEntityAccess", Value: false}` unconditionally, +// so a microflow that ran under the user's entity access rules came back running +// with full access. Nothing reported it: the model is valid either way. +func TestMicroflowApplyEntityAccessRoundTrip(t *testing.T) { + for _, want := range []bool{true, false} { + mf := µflows.Microflow{Name: "ACT_Secured", ApplyEntityAccess: want} + mf.ID = model.ID("mf-1") + + w := testWriter() + raw, err := w.serializeMicroflow(mf) + if err != nil { + t.Fatalf("serialize: %v", err) + } + var doc map[string]any + if err := bson.Unmarshal(raw, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if got, ok := doc["ApplyEntityAccess"].(bool); !ok || got != want { + t.Errorf("written ApplyEntityAccess = %#v, want %v", doc["ApplyEntityAccess"], want) + } + + // And the parser has to read it back, or the value never reaches the + // writer on a rewrite in the first place. + back := ParseMicroflowFromRaw(doc, "mf-1", "mod-1") + if back.ApplyEntityAccess != want { + t.Errorf("parsed ApplyEntityAccess = %v, want %v", back.ApplyEntityAccess, want) + } + } +} From 4ed9cd4bac737198b6388276233636964f91c50c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 05:27:39 +0000 Subject: [PATCH 06/18] fix(mdl): MDL059 now covers annotations written before a CREATE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An annotation the document does not read parsed, executed, and built at 0 errors with the annotation dropped. Three shapes, all real: @applyentityacces -- a typo; the security setting is simply not applied @applyentityaccess -- on a NANOFLOW, which has no such property @caption -- an activity annotation written at document level The second was created by the change that added @applyentityaccess and left open with the note that it fails safe. Failing safe is not being reported. ## Why there was nowhere for the check to live MDL's grammar attaches `annotation*` to `createStatement` ITSELF, so all forty-odd create kinds accept an annotation while only six read one. Each document's builder looks for the names it implements, so a name NO builder implements is invisible to every one of them. So the check goes where the parse tree is: `ExitCreateStatement` records every annotation with the kind of document it was written on, and one validator owns the policy. The kind is derived from the grammar's own rule names (createMicroflowStatement -> "microflow") rather than forty type assertions, which means a create statement added later is covered the day it is added — defaulting to "reads no annotations" instead of being silently forgotten. MDL059 rather than a new rule: it is the same defect the statement-level rule already refuses, and #884's reasoning is unchanged. Both `check` and `exec` run ValidateProgram, so the two cannot drift. ## Pinned in both directions The accepted set is a per-kind table beside knownActivityAnnotations, held to the visitor's own string literals by a test that fails BOTH ways: a name the visitor reads but the table omits would reject a valid script, and a name the table lists but nothing reads would re-open the hole. That AST scrape needs one non-obvious filter — match only EqualFold calls whose other operand mentions AnnotationName, or it also collects the literal "false" from applyEntityAccessAnnotation testing its own VALUE. ## Blast radius, measured before erroring on what used to be silent A scan of every .mdl and every skill block found exactly the seven implemented combinations and no stray annotation. `make check-mdl`, `make check-skill-mdl` and the full suite are unchanged. Without that measurement, turning silence into an error would have been a guess. Control: unwiring the rule makes mdl-examples/bug-tests/document-annotation-typos.fail.mdl pass check again. That file carries all three refusals plus a valid microflow, so it would not pass against a rule that simply refused every document annotation. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../write-microflows/reference/pitfalls.md | 4 +- cmd/mxcli/syntax/features_microflow.go | 5 +- docs/01-project/MDL_QUICK_REFERENCE.md | 1 + .../microflow-rewrite-property-audit.md | 19 +- .../document-annotation-typos.fail.mdl | 51 ++++ mdl/ast/ast.go | 27 +++ mdl/executor/validate_document_annotations.go | 103 +++++++++ .../validate_document_annotations_test.go | 217 ++++++++++++++++++ mdl/executor/validate_program.go | 6 + mdl/visitor/visitor.go | 8 +- mdl/visitor/visitor_document_annotations.go | 92 ++++++++ 12 files changed, 526 insertions(+), 8 deletions(-) create mode 100644 mdl-examples/bug-tests/document-annotation-typos.fail.mdl create mode 100644 mdl/executor/validate_document_annotations.go create mode 100644 mdl/executor/validate_document_annotations_test.go create mode 100644 mdl/visitor/visitor_document_annotations.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index d6a305bf6..79372745c 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -589,3 +589,4 @@ {"area": "mdl/executor", "date": "2026-09-11", "symptom": "A DESCRIBE-side resolver for SOAP references had been unreachable since it was written, and its unit test passed. `describe microflow` printed the right service and mapping names throughout", "cause": "`resolveWebServiceReference` looked the service up by ELEMENT ID among units of type `WebServices$ImportedWebService`. It could not match on two independent counts: the stored $Type is `WebServices$ImportedServiceImpl` (ImportedWebService is the SDK name, and nothing is stored under it), and the value compared was never an id \u2014 ImportedService is a BY_NAME_REFERENCE, so it already held `Clients.OrderSoapClient`. Same for the two mapping resolvers. Every call fell through to a fallback that returned the stored string, which is the correct answer", "file": "`mdl/executor/cmd_microflows_format_action.go` (formatWebServiceCallAction; the five resolvers removed)", "insight": "**A resolver whose fallback is the correct answer is indistinguishable in its output from one that works \u2014 so only the input side can prove it runs.** Nothing in the DESCRIBE text could ever have been wrong, which is why this survived: the test that covered it, `TestFormatAction_WebServiceCallResolvesKnownReferences`, built a world where ServiceID WAS a unit id and the unit type WAS `ImportedWebService`, neither of which occurs in any project \u2014 the green test asserted the fiction, not the code. The replacement inverts it: the mock backend calls `t.Fatal` if it is consulted at all, so the test fails unless no lookup happens. **The wider fact, measured, is that the structured DESCRIBE branch is unreachable for real SOAP calls anyway**: all three of ako/TestApp's carry 15 keys and `webServiceActionRequiresRawBSON` supports 9, and mxcli's own writer emits the same 15, so every SOAP call on either engine describes as `call web service raw ''`. A branch nothing reaches cannot be validated by any amount of passing tests over it. **Check a BY_NAME_REFERENCE before writing a resolver**: `modelsdk/gen/*/refs.go` states the kind (`codec.RefByName` here), and an existing test two packages away was already passing `Mod.Service` as the value", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "`send mapping X` on a SOAP call parsed, `exec` reported success, and the mapping name appeared ZERO times in the written document \u2014 on both engines. mxbuild then rejected the call as `[CE0369] \"Cannot use simple request body, as the operation's body is complex\"`. Separately, an operation taking parameters had no MDL syntax at all and built as `[CE0178] \"Body parameter mapping needs to be refreshed.\"`", "cause": "Both writers emitted an unconditional empty `Microflows$SimpleRequestHandling` for `RequestBodyHandling`. It is a POLYMORPHIC child holding EITHER the operation's arguments (SimpleRequestHandling + WebServiceOperationSimpleParameterMapping entries) or an export mapping (`Microflows$MappingRequestHandling`, NOT the `Mendix$AdvancedRequestHandling` legacy's comment named \u2014 that type is in none of the three ako/TestApp reference documents). The reader compounded it by looking for `RequestHandling`/`ExportMappingCall`, keys no real document carries; the stored key is `RequestBodyHandling`", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go`, `sdk/mpr/writer_microflow_actions.go` (webServiceRequestBody), `mdl/executor/webservice_names.go` (webServiceParameterPath), `mdl/grammar/domains/MDLMicroflow.g4`", "insight": "**Two gaps that look independent can be one polymorphic property, and finding that out is what makes the validation possible.** Arguments and the send mapping are the two branches of `RequestBodyHandling`, so the mutual-exclusion rule (MDL-SOAP01) only becomes enforceable once BOTH exist \u2014 implementing either alone means `check` can refuse a combination it cannot offer an alternative to. **The stored ParameterPath is derivable, which is what makes readable syntax possible**: `escape(operation.RequestBodyElementName) + \"|\" + name` reads the element off `Description.Services[].Operations[]` of the imported service, so MDL says `OrderId` and not `http%3A//www.example.com/:GetOrder|OrderId`. Escaping is per SEGMENT \u2014 `:` inside a segment becomes %3A, the separator `:` and the `/`es are left alone \u2014 and a segment containing `%` is REFUSED because Mendix's escaping of it is unmeasured. **The typed-array marker is the silent trap**: a populated ParameterMappings list leads with marker 2 and `codec.lookupListMarker` DEFAULTS TO 3, so without a `RegisterListMarker` the arguments serialize under the wrong array version \u2014 invisible to mxbuild, fatal to Studio Pro. Control: stubbing the registration makes the test report `marker = 3, want int32(2)`", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec`, the documented copy operation, silently turned OFF a microflow's \"apply entity access\" (16/342 microflows across 4 projects, 11.14.0) and turned a blocking `show message` into a non-blocking one (16 microflows). Neither `mxcli check` nor mxbuild says anything: the model is valid either way, and the only consequence is that a constrained user behaves differently", "cause": "TWO causes, one symptom. ApplyEntityAccess was HARDCODED false in both writers and `microflows.Microflow` had no field, so the read side dropped it first \u2014 while `microflows.Rule` carried it correctly in the same codebase (and its CREATE path never set it, so rules had the same bug from the other end). ShowMessageAction.Blocking was carried perfectly by BOTH engines; MDL simply had no keyword, so DESCRIBE could not emit it and the re-parse set false", "file": "`sdk/microflows/microflows.go`, `mdl/backend/modelsdk/microflow.go`+`microflow_write.go`, `sdk/mpr/parser_microflow.go`+`writer_microflow.go`, `mdl/executor/apply_entity_access.go`, `mdl/executor/cmd_microflows_build.go`, `mdl/executor/cmd_rules_create.go`, `mdl/grammar/MDLLexer.g4` (BLOCKING)", "insight": "**Carrying a property through the writers is only half a round-trip fix when the middle is MDL text.** describe -> exec rebuilds from the STATEMENT, so a setting the statement never names is gone however well the storage layer handles it \u2014 which is why ApplyEntityAccess needed BOTH preserve-on-rewrite (for CREATE OR REPLACE) and an annotation (for the copy case, where there is nothing to preserve from). **Absent must not mean false for a security setting**: the AST field is a *bool so `@applyentityaccess(false)` and silence are distinguishable, the same rule @excluded (#914) and the doc comment (#1018) already follow. No grammar change was needed \u2014 `annotationValue` already accepts a literal, so a parameterised annotation is free. **Adding a lexer keyword needs a keyword-rule entry and a test**: BLOCKING would otherwise stop `blocking` being usable as a parameter name. **The fix is only believable with the audit re-run as its verification** \u2014 three unit-test controls plus the same 342-microflow measurement showing both properties stopped moving. That re-run also surfaced CE0709 \"Sequence flow is not accepted by origin or destination\" on a whole-project round trip, which the pre-fix binary reproduces identically: a separate, pre-existing flow-graph defect that would have been easy to misattribute to this change", "refs": []} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "An annotation written before a CREATE that the document does not read \u2014 a typo (`@applyentityacces`), the right name on the wrong document kind (`@applyentityaccess` on a nanoflow), or an activity annotation at document level (`@caption`) \u2014 passed `mxcli check`, passed `exec`, and built at 0 errors with the annotation silently dropped", "cause": "MDL's grammar attaches `annotation*` to `createStatement` ITSELF, so all forty-odd create kinds accept an annotation while only six read one. Each document's builder looks only for the names it implements, so a name NO builder implements is invisible to all of them \u2014 there was nowhere the check could have lived. MDL059 already refused exactly this one node family over (on statements), and #884's reasoning applies unchanged", "file": "`mdl/visitor/visitor_document_annotations.go` (ExitCreateStatement), `mdl/executor/validate_document_annotations.go`, `mdl/ast/ast.go` (Program.DocumentAnnotations)", "insight": "**When a check has no natural home in any single handler, the parse tree is the home.** Deriving the document kind from the grammar's OWN rule names (`parser.MDLParserParserStaticData.RuleNames`, `createMicroflowStatement` -> \"microflow\") beats forty type assertions and means a create statement added later is covered the day it is added, defaulting to 'reads no annotations' rather than being silently forgotten. **Record everything, decide centrally**: the visitor logs every document annotation with its kind and the validator owns the policy, so the accepted set is one table beside `knownActivityAnnotations` instead of being spread across the seven visitor sites. **Pin the table to the visitor in BOTH directions** \u2014 a name the visitor reads but the table omits rejects a valid script; a name the table lists but nothing reads re-opens the hole. The AST-scraping test that does this needs one non-obvious filter: match only EqualFold calls whose other operand mentions `AnnotationName`, or it also collects the literal \"false\" from an annotation that compares its own VALUE. **Measure the blast radius before erroring on something previously silent**: a scan of every .mdl and skill block found exactly the seven implemented combinations and no stray annotation, and both MDL corpora plus the full suite stayed green \u2014 without that, turning silence into an error is a guess", "refs": []} diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index 728e11dd9..ad456e677 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -535,4 +535,6 @@ It is a **security** setting and it only ever narrows, so the rules mirror storing the flag came back without it, and nothing anywhere reported it. - **Turning it off is explicit**: `@applyentityaccess(false)`. - **Not available on a nanoflow.** A nanoflow runs in the client and Mendix stores - no such property, so the annotation would parse and do nothing. + no such property. Writing it there is **MDL059**, not a silent no-op — + the same rule that catches `@applyentityacces` and any other annotation the + document does not read. The message names what that document does accept. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 1f2c85ee6..3b289a043 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -308,7 +308,10 @@ func init() { "@annotation(id: n1, text: 'a note', position: (x, y), size: (w, h))\n" + "@annotation(id: n1) -- attaches THAT note to another activity\n\n" + "An unrecognised @name is an error (MDL059): it would parse and do nothing,\n" + - "so a typo of @position would silently discard the layout.\n\n" + + "so a typo of @position would silently discard the layout. That covers\n" + + "DOCUMENT annotations too — a typo, or one on a document kind that does not\n" + + "read it (@applyentityaccess on a nanoflow, @excluded on a queue), is\n" + + "refused with the list of what that document does accept.\n\n" + "@excluded and @applyentityaccess are DOCUMENT annotations — they go before\n" + "CREATE, not on a statement. @applyentityaccess runs the flow under the\n" + "current user's entity access rules instead of with full access; it is a\n" + diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 071016cfb..81e484a5a 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -545,6 +545,7 @@ it is for pages. | Log | `log info\|warning\|error [node 'name'] 'message';` | | | Apply entity access | `@applyentityaccess` / `@applyentityaccess(false)` before `create microflow` or `create rule` | Runs the flow under the **current user's** entity access rules instead of with full access. A **security** setting and only ever narrowing, so an ABSENT annotation **preserves** what is stored rather than clearing it — the same rule as `@excluded`. Not available on a nanoflow: it runs in the client and Mendix stores no such property | | Position | `@position(x, y)` | Canvas position (before activity) | +| Unknown annotation | — | **MDL059**. An annotation that parses and does nothing loses whatever it was meant to express, so a name the target does not read is refused — on a statement *and* before a `create`. Covers a typo (`@applyentityacces`), an annotation on a document kind that reads none (`@excluded` on a queue), and an activity annotation written at document level. The message names what that document does accept | | Parameter position | `@position(x, y)` before a parameter, **inside** the `( … )` list | The only annotation a parameter takes. Omit it and parameters form a row at 200;53, 300;53, …; a parameter off that row is treated as hand-placed, survives a rewrite, and is emitted by DESCRIBE (#993) | | Start event | `@start(x, y)` | Canvas position of the start, on the **first** statement. Omit it and the start is placed one spacing unit left of the first activity and MOVES with it on a rewrite; a start that is not at that derived spot is treated as hand-placed, survives a rewrite, and is emitted by DESCRIBE (#951) | | Caption | `@caption 'text'` | Custom caption (before activity) | diff --git a/docs/12-bug-reports/microflow-rewrite-property-audit.md b/docs/12-bug-reports/microflow-rewrite-property-audit.md index e46aa89a7..0901c2b9d 100644 --- a/docs/12-bug-reports/microflow-rewrite-property-audit.md +++ b/docs/12-bug-reports/microflow-rewrite-property-audit.md @@ -136,11 +136,20 @@ than the 417-line diff, and wants its own investigation. Neither is a demonstrated loss (see the benign table), so closing those holes needs a reference project that actually sets them. -One gap this change does not close: a **typo'd document annotation** parses and -does nothing. MDL059 covers statement annotations only, so `@applyentityacces` -is silent — as `@excluded` already was. Both typos fail safe here (an unset flag -on a create means off; on a rewrite it means preserve), which is why this is -noted rather than fixed. +~~One gap this change does not close: a typo'd document annotation parses and does +nothing.~~ **Closed.** MDL059 now covers annotations written before a `CREATE` +as well as those on a statement. The grammar attaches `annotation*` to +`createStatement` itself, so all forty-odd create kinds accepted one while only +six read one — a typo (`@applyentityacces`), an annotation on a kind that reads +none (`@excluded` on a queue), and an activity annotation written at document +level all parsed, executed and built at 0 errors with the annotation dropped. + +The accepted set is a per-kind table pinned to the visitor's own string literals +in both directions, so a name the visitor reads but the table omits (which would +reject a valid script) and a name the table lists but nothing reads (which would +re-open the hole) each fail a test. `mdl-examples/bug-tests/document-annotation-typos.fail.mdl` +carries all three refusals plus a valid microflow as its control; unwiring the +rule makes that file pass check again. ## Method, and what it is worth diff --git a/mdl-examples/bug-tests/document-annotation-typos.fail.mdl b/mdl-examples/bug-tests/document-annotation-typos.fail.mdl new file mode 100644 index 000000000..0b7101b40 --- /dev/null +++ b/mdl-examples/bug-tests/document-annotation-typos.fail.mdl @@ -0,0 +1,51 @@ +-- Document annotations that parse and do nothing (refusals, MDL059). +-- +-- `mxcli check mdl-examples/bug-tests/document-annotation-typos.fail.mdl` +-- reports MDL059 three times. Before the fix all three passed check AND exec, +-- and wrote a model that built at 0 errors — the annotation was simply dropped, +-- so whatever it was meant to express was lost in silence. +-- +-- The grammar attaches `annotation*` to createStatement itself, so all forty-odd +-- create kinds accept one while only six read one. That is the whole defect: +-- being accepted by the parser said nothing about being read. +-- +-- The valid microflow at the end is the control. Without it this file would pass +-- just as well against a rule that refused every document annotation. + +create or modify module AnnProbe; + +-- 1. A TYPO. One 's' short, and the security setting is simply not applied: +-- the microflow runs with full access instead of the current user's. +@applyentityacces +create or replace microflow AnnProbe.Typo () +begin + return; +end; +/ + +-- 2. The right annotation on the WRONG DOCUMENT. A nanoflow runs in the client +-- and Nanoflows$Nanoflow stores no such property, so there is nothing for it +-- to set. This one was created by the change that added the annotation. +@applyentityaccess +create or replace nanoflow AnnProbe.WrongKind () +begin + return; +end; +/ + +-- 3. An ACTIVITY annotation written at document level. @caption belongs on a +-- statement inside the body; before CREATE it reaches nothing. +@caption 'My microflow' +create or replace microflow AnnProbe.ActivityAnnotationOutside () +begin + return; +end; +/ + +-- CONTROL: both annotations a microflow really does read. This must stay clean. +@excluded +@applyentityaccess +create or replace microflow AnnProbe.Fine () +begin + return; +end; diff --git a/mdl/ast/ast.go b/mdl/ast/ast.go index ad2910b3a..5c4aa3257 100644 --- a/mdl/ast/ast.go +++ b/mdl/ast/ast.go @@ -38,6 +38,33 @@ func (q QualifiedName) String() string { // Program represents a complete MDL program (sequence of statements). type Program struct { Statements []Statement + // DocumentAnnotations records every annotation written before a CREATE — + // `@excluded`, `@position`, `@applyentityaccess` and anything misspelled — + // paired with the kind of document it was written on. + // + // The grammar lets ANY create statement carry annotations while only seven + // document kinds read one, so an annotation on the wrong document, or with a + // typo in it, parsed and did nothing. That is the failure MDL059 already + // refuses one node family over: whatever the annotation was meant to express + // is lost in silence. + // + // Every annotation is recorded rather than only the unrecognised ones, so + // which names a document accepts stays a single decision in the validator + // next to knownActivityAnnotations, instead of being spread across the seven + // visitor sites that read them. + DocumentAnnotations []DocumentAnnotation +} + +// DocumentAnnotation is one annotation written before a CREATE statement. +type DocumentAnnotation struct { + // Kind is the document it was written on, in MDL's own words ("microflow", + // "nanoflow", "entity", …), so the message can name it. + Kind string + // Name is the annotation, lower-cased and without the "@". + Name string + // Target is the document's qualified name where the visitor could read one, + // for a message that points at the right statement in a long script. + Target string } // ============================================================================ diff --git a/mdl/executor/validate_document_annotations.go b/mdl/executor/validate_document_annotations.go new file mode 100644 index 000000000..634d08954 --- /dev/null +++ b/mdl/executor/validate_document_annotations.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation for annotations written before a CREATE. +// +// MDL's grammar attaches `annotation*` to createStatement itself, so all forty-odd +// create kinds accept one while only six read one. Everything else parsed and did +// nothing — the failure MDL059 already refuses one node family over, and the same +// one #884 was filed for: an annotation that parses and does nothing loses +// whatever it was meant to express, silently. +// +// Two shapes it lets through, both real: +// +// @applyentityacces -- a typo; the security setting is simply not applied +// @applyentityaccess -- on a NANOFLOW, which has no such property at all +// +// The second is not hypothetical: it was created by the change that added the +// annotation, and left open with the note that it fails safe. Failing safe is not +// the same as being reported. +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// documentAnnotations maps a document kind to the annotations its builder reads. +// A kind absent from this map reads none, which is the honest default: the +// grammar accepts an annotation there and nothing acts on it. +// +// Keys are the kind names ExitCreateStatement derives from the grammar's own rule +// names, so they track the grammar rather than a hand-kept list. Values are the +// names each visitor actually tests for — TestDocumentAnnotationsMatchTheVisitor +// pins the two together, because a name added to one and not the other either +// rejects a valid annotation or silently drops an invalid one. +var documentAnnotations = map[string]map[string]bool{ + // visitor_entity.go — @position places the entity on the domain-model canvas. + // Covers view entities too: both go through createEntityStatement. + "entity": {"position": true}, + // visitor_association.go — @anchor picks which side each end attaches to. + "association": {"anchor": true}, + // visitor_microflow.go + "microflow": {"excluded": true, "applyentityaccess": true}, + "nanoflow": {"excluded": true}, + "rule": {"excluded": true, "applyentityaccess": true}, + // visitor_page_v3.go + "page": {"excluded": true}, +} + +// ValidateDocumentAnnotations reports (MDL059) an annotation the document it is +// written on does not read. +func ValidateDocumentAnnotations(prog *ast.Program) []linter.Violation { + var out []linter.Violation + for _, ann := range prog.DocumentAnnotations { + if ann.Kind == "" { + continue // no create statement parsed — a syntax error already reported + } + if documentAnnotations[ann.Kind][ann.Name] { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL059", + Severity: linter.SeverityError, + Message: documentAnnotationMessage(ann), + Suggestion: documentAnnotationSuggestion(ann.Kind), + }) + } + return out +} + +func documentAnnotationMessage(ann ast.DocumentAnnotation) string { + target := "" + if ann.Target != "" { + target = " on " + ann.Target + } + return fmt.Sprintf("unknown annotation `@%s`%s — a %s does not read it, so it "+ + "parses and does nothing and whatever it was meant to express is silently lost", + ann.Name, target, ann.Kind) +} + +// documentAnnotationSuggestion names what the document DOES accept, so a typo is +// one line from being fixed — and so the nanoflow case reads as the real answer +// it is rather than as a bare refusal. +func documentAnnotationSuggestion(kind string) string { + accepted := documentAnnotations[kind] + if len(accepted) == 0 { + return fmt.Sprintf("A %s reads no annotations at all. Annotations before CREATE "+ + "belong to entity (@position), association (@anchor), page/nanoflow (@excluded) "+ + "and microflow/rule (@excluded, @applyentityaccess); activity annotations "+ + "(@position, @caption, @colour, …) go inside the flow body, on the statement "+ + "they belong to.", kind) + } + names := make([]string, 0, len(accepted)) + for name := range accepted { + names = append(names, "@"+name) + } + sort.Strings(names) + return fmt.Sprintf("A %s reads %s. If this is a typo of one of those, correct it; "+ + "otherwise remove it.", kind, strings.Join(names, " and ")) +} diff --git a/mdl/executor/validate_document_annotations_test.go b/mdl/executor/validate_document_annotations_test.go new file mode 100644 index 000000000..53b57f81f --- /dev/null +++ b/mdl/executor/validate_document_annotations_test.go @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + goast "go/ast" + goparser "go/parser" + "go/token" + "sort" + "strconv" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// visitorFilesWithDocumentAnnotations are the visitor sources that read an +// annotation off a createStatement. A new one has to be listed here, which the +// union check below then holds to the table. +var visitorFilesWithDocumentAnnotations = []string{ + "../visitor/visitor_association.go", + "../visitor/visitor_entity.go", + "../visitor/visitor_microflow.go", + "../visitor/visitor_page_v3.go", +} + +// TestDocumentAnnotationsMatchTheVisitor pins the table to the visitor's own +// string literals, in BOTH directions. +// +// A name the visitor reads but the table omits rejects a valid script. A name the +// table lists but no visitor reads accepts an annotation that does nothing — the +// very thing this rule exists to catch. Neither is visible without comparing the +// two, which is why they are compared rather than kept in step by hand. +func TestDocumentAnnotationsMatchTheVisitor(t *testing.T) { + read := map[string]bool{} + fset := token.NewFileSet() + for _, path := range visitorFilesWithDocumentAnnotations { + file, err := goparser.ParseFile(fset, path, nil, 0) + if err != nil { + t.Fatalf("parse %s: %v", path, err) + } + goast.Inspect(file, func(n goast.Node) bool { + fn, ok := n.(*goast.FuncDecl) + if !ok || fn.Body == nil || !mentionsAllAnnotation(fn) { + return true + } + for _, name := range equalFoldLiterals(fn.Body) { + read[strings.ToLower(name)] = true + } + return false + }) + } + if len(read) == 0 { + t.Fatal("found no annotation names in the visitor; this test no longer measures anything") + } + + listed := map[string]bool{} + for _, names := range documentAnnotations { + for name := range names { + listed[name] = true + } + } + if missing := difference(read, listed); len(missing) > 0 { + t.Errorf("the visitor reads %v but documentAnnotations does not list them — "+ + "MDL059 would reject a valid script", missing) + } + if extra := difference(listed, read); len(extra) > 0 { + t.Errorf("documentAnnotations lists %v but no visitor reads them — the annotation "+ + "would parse and do nothing, unreported", extra) + } +} + +// mentionsAllAnnotation reports whether a function reads a createStatement's +// annotations, which is what makes it a document-annotation site. +func mentionsAllAnnotation(fn *goast.FuncDecl) bool { + found := false + goast.Inspect(fn.Body, func(n goast.Node) bool { + if sel, ok := n.(*goast.SelectorExpr); ok && sel.Sel.Name == "AllAnnotation" { + found = true + } + return !found + }) + return found +} + +// equalFoldLiterals collects the string literals compared against an annotation's +// NAME with strings.EqualFold — how every document-annotation site spells its +// test. +// +// The other operand must mention AnnotationName. Without that filter the walk +// also picks up EqualFold calls on an annotation's VALUE — applyEntityAccessAnnotation +// tests its parameter against "false" — and "false" is not an annotation name. +func equalFoldLiterals(body *goast.BlockStmt) []string { + var out []string + goast.Inspect(body, func(n goast.Node) bool { + call, ok := n.(*goast.CallExpr) + if !ok { + return true + } + sel, ok := call.Fun.(*goast.SelectorExpr) + if !ok || sel.Sel.Name != "EqualFold" || !mentionsAnnotationName(call) { + return true + } + for _, arg := range call.Args { + lit, ok := arg.(*goast.BasicLit) + if !ok || lit.Kind != token.STRING { + continue + } + if s, err := strconv.Unquote(lit.Value); err == nil { + out = append(out, s) + } + } + return true + }) + return out +} + +// mentionsAnnotationName reports whether a call reads an annotation's name — +// either directly (ann.AnnotationName().GetText()) or through a local holding it +// (annName), which two of the entity sites use. +func mentionsAnnotationName(call *goast.CallExpr) bool { + found := false + goast.Inspect(call, func(n goast.Node) bool { + switch t := n.(type) { + case *goast.SelectorExpr: + if t.Sel.Name == "AnnotationName" { + found = true + } + case *goast.Ident: + if t.Name == "annName" { + found = true + } + } + return !found + }) + return found +} + +func difference(a, b map[string]bool) []string { + var out []string + for k := range a { + if !b[k] { + out = append(out, k) + } + } + sort.Strings(out) + return out +} + +// TestValidateDocumentAnnotations covers the three shapes that used to be +// silent, and the one that must stay silent. +func TestValidateDocumentAnnotations(t *testing.T) { + for _, tc := range []struct { + name string + ann ast.DocumentAnnotation + want bool // want a violation + }{ + // The typo. Costs the security setting and reports nothing. + {"typo on a microflow", ast.DocumentAnnotation{Kind: "microflow", Name: "applyentityacces", Target: "M.F"}, true}, + // The right annotation on a document with no such property — created by + // the change that added @applyentityaccess, and left open at the time. + {"right name, wrong document", ast.DocumentAnnotation{Kind: "nanoflow", Name: "applyentityaccess"}, true}, + // A document kind that reads no annotation at all. + {"annotation on a queue", ast.DocumentAnnotation{Kind: "queue", Name: "excluded"}, true}, + // An ACTIVITY annotation written before CREATE instead of inside the body. + {"activity annotation at document level", ast.DocumentAnnotation{Kind: "microflow", Name: "caption"}, true}, + + {"excluded on a microflow", ast.DocumentAnnotation{Kind: "microflow", Name: "excluded"}, false}, + {"applyentityaccess on a rule", ast.DocumentAnnotation{Kind: "rule", Name: "applyentityaccess"}, false}, + {"position on an entity", ast.DocumentAnnotation{Kind: "entity", Name: "position"}, false}, + {"anchor on an association", ast.DocumentAnnotation{Kind: "association", Name: "anchor"}, false}, + {"excluded on a page", ast.DocumentAnnotation{Kind: "page", Name: "excluded"}, false}, + // A create statement that did not parse has no kind, and a syntax error + // is already reported for it — a second complaint would be noise. + {"unparsed create statement", ast.DocumentAnnotation{Name: "excluded"}, false}, + } { + t.Run(tc.name, func(t *testing.T) { + got := ValidateDocumentAnnotations(&ast.Program{ + DocumentAnnotations: []ast.DocumentAnnotation{tc.ann}, + }) + if tc.want && len(got) == 0 { + t.Fatalf("@%s on a %s was accepted", tc.ann.Name, tc.ann.Kind) + } + if !tc.want { + if len(got) != 0 { + t.Fatalf("@%s on a %s was rejected: %s", tc.ann.Name, tc.ann.Kind, got[0].Message) + } + return + } + if got[0].RuleID != "MDL059" || got[0].Severity != linter.SeverityError { + t.Errorf("violation = %s/%v, want MDL059/error", got[0].RuleID, got[0].Severity) + } + // The message has to name the document kind, or a reader cannot tell + // a typo from an annotation on the wrong document. + if !strings.Contains(got[0].Message, tc.ann.Kind) { + t.Errorf("message does not name the document kind: %s", got[0].Message) + } + }) + } +} + +// TestDocumentAnnotationSuggestionNamesTheAlternatives — the suggestion has to +// say what the document DOES take, so the nanoflow case reads as an answer +// rather than a bare refusal. +func TestDocumentAnnotationSuggestionNamesTheAlternatives(t *testing.T) { + if got := documentAnnotationSuggestion("microflow"); !strings.Contains(got, "@applyentityaccess") || + !strings.Contains(got, "@excluded") { + t.Errorf("microflow suggestion = %q", got) + } + if got := documentAnnotationSuggestion("nanoflow"); !strings.Contains(got, "@excluded") { + t.Errorf("nanoflow suggestion = %q", got) + } + if got := documentAnnotationSuggestion("queue"); !strings.Contains(got, "no annotations at all") { + t.Errorf("queue suggestion = %q", got) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 0f89671ab..14d3cc650 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -253,5 +253,11 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // passed check. violations = append(violations, ValidateScheduledEvents(prog)...) + // Flag an annotation written before a CREATE that the document does not + // read — a typo, or one on the wrong document kind. The grammar accepts an + // annotation on every create statement while only six read one, so these + // parsed and did nothing (MDL059, the same rule statements already have). + violations = append(violations, ValidateDocumentAnnotations(prog)...) + return violations } diff --git a/mdl/visitor/visitor.go b/mdl/visitor/visitor.go index 6b3f68ffd..dd07a7c78 100644 --- a/mdl/visitor/visitor.go +++ b/mdl/visitor/visitor.go @@ -464,6 +464,9 @@ type Builder struct { program *ast.Program statements []ast.Statement errors []error + // documentAnnotations collects every `@name` written before a CREATE, with + // the kind of document it was on — see ExitCreateStatement. + documentAnnotations []ast.DocumentAnnotation } // NewBuilder creates a new AST builder. @@ -521,7 +524,10 @@ func Build(input string) (*ast.Program, []error) { // Combine syntax errors and builder errors allErrors := append(errListener.errors, builder.errors...) - return &ast.Program{Statements: builder.statements}, allErrors + return &ast.Program{ + Statements: builder.statements, + DocumentAnnotations: builder.documentAnnotations, + }, allErrors } // Errors returns any errors encountered during building. diff --git a/mdl/visitor/visitor_document_annotations.go b/mdl/visitor/visitor_document_annotations.go new file mode 100644 index 000000000..91fcf805c --- /dev/null +++ b/mdl/visitor/visitor_document_annotations.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + + "github.com/antlr4-go/antlr/v4" +) + +// ExitCreateStatement records every annotation written before a CREATE, paired +// with the kind of document it was written on. +// +// The grammar attaches `annotation*` to createStatement itself, so ANY of the +// forty-odd create kinds accepts one while only seven read one. An annotation on +// a document that does not read it — or with a typo in it — therefore parsed and +// did nothing, which is exactly the failure MDL059 refuses one node family over. +// +// Recording happens here rather than in each document's own builder for two +// reasons: those builders only look for the names they implement, so a name none +// of them implements is invisible to all of them; and deriving the kind from the +// parse tree means a create statement added later is covered without anyone +// remembering to add it. +func (b *Builder) ExitCreateStatement(ctx *parser.CreateStatementContext) { + if ctx == nil { + return + } + anns := ctx.AllAnnotation() + if len(anns) == 0 { + return + } + kind := createStatementKind(ctx) + for _, a := range anns { + annCtx, ok := a.(*parser.AnnotationContext) + if !ok || annCtx.AnnotationName() == nil { + continue + } + b.documentAnnotations = append(b.documentAnnotations, ast.DocumentAnnotation{ + Kind: kind, + Name: strings.ToLower(annCtx.AnnotationName().GetText()), + Target: createStatementTarget(ctx), + }) + } +} + +// createStatementKind names the document a create statement builds, in MDL's own +// words: "microflow", "entity", "scheduledevent". +// +// It reads the child rule's own name out of the parser's rule table rather than +// testing forty accessors, so a create statement added to the grammar is named +// correctly here the day it is added. "" when no child rule is present, which +// only happens on a parse error. +func createStatementKind(ctx *parser.CreateStatementContext) string { + for _, child := range ctx.GetChildren() { + rc, ok := child.(antlr.RuleContext) + if !ok { + continue + } + names := parser.MDLParserParserStaticData.RuleNames + idx := rc.GetRuleIndex() + if idx < 0 || idx >= len(names) { + continue + } + name := names[idx] + if !strings.HasPrefix(name, "create") || !strings.HasSuffix(name, "Statement") { + continue // docComment / annotation, not the document itself + } + return strings.ToLower(strings.TrimSuffix(strings.TrimPrefix(name, "create"), "Statement")) + } + return "" +} + +// createStatementTarget is the first qualified name inside the create statement, +// used only to point the message at the right statement in a long script. +func createStatementTarget(ctx *parser.CreateStatementContext) string { + var walk func(antlr.Tree) string + walk = func(n antlr.Tree) string { + if qn, ok := n.(*parser.QualifiedNameContext); ok { + return qn.GetText() + } + for _, c := range n.GetChildren() { + if got := walk(c); got != "" { + return got + } + } + return "" + } + return walk(ctx) +} From dd50f106b1f2e7bddc76e05abf565d63b2f9b57c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 05:32:17 +0000 Subject: [PATCH 07/18] Make `Action: NOTHING` a real action expression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `NOTHING` is the documented spelling for a deliberately inert widget — a decorative button, a card that is styled but not clickable. It appears in docs-site, MDL_QUICK_REFERENCE.md, the synced alter-page skill and nine mdl-examples scripts, and it was never in the grammar. It worked by accident. `actionExprV3` had no NOTHING alternative, so the property fell through to the generic `keyword COLON propertyValueV3` branch at the end of widgetPropertyV3, the slot ended up holding a plain string, GetAction() type-asserted and returned nil, and the writer's default branch emitted Forms$NoAction — which is exactly what the author wanted. That same fall-through is mendixlabs/mxcli#1062: `Action: OPEN_LINK` with no URL, and `Action: TOTALLY_MADE_UP`, take the identical route to the identical dead widget, silently. The scalar cannot be rejected while the documented form still depends on it, so this promotion comes first. Nothing changes on disk. The form maps to the existing pages.NoClientAction, which every backend already treats as nil, and a page written by the pre-fix binary is reported `Unchanged page` when the post-fix binary re-runs the same script — the canonical comparison in ADR-0008 saying the documents are semantically identical. Verified end to end on Mendix 11.12.0: exec, describe round-trip, and `mx check` at 0 errors. buildClientActionV3's switch ends in a `default:` that refuses unknown action types, so the grammar and the builder have to move together: promoting the form without its case would turn a working spelling into an exec failure. Removing the grammar alternative does not even compile (`actCtx.NOTHING undefined`), which is the control for that coupling. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JEZmExJUvn2nWTWE9mrd4i --- .../doctype-tests/03-page-examples.mdl | 32 +++++++++ mdl/executor/cmd_fragments.go | 2 + .../cmd_pages_builder_action_nothing_test.go | 52 ++++++++++++++ mdl/executor/cmd_pages_builder_v3.go | 14 ++++ mdl/grammar/domains/MDLPage.g4 | 14 ++++ .../visitor_page_action_nothing_test.go | 70 +++++++++++++++++++ mdl/visitor/visitor_page_v3.go | 7 ++ 7 files changed, 191 insertions(+) create mode 100644 mdl/executor/cmd_pages_builder_action_nothing_test.go create mode 100644 mdl/visitor/visitor_page_action_nothing_test.go diff --git a/mdl-examples/doctype-tests/03-page-examples.mdl b/mdl-examples/doctype-tests/03-page-examples.mdl index b1bfa17bf..cf2c5da23 100644 --- a/mdl-examples/doctype-tests/03-page-examples.mdl +++ b/mdl-examples/doctype-tests/03-page-examples.mdl @@ -2818,3 +2818,35 @@ create page PgTest.P_OrderWithCustomer ( } } } + +-- ============================================================================ +-- MARK: An explicitly inert widget — `action: nothing` +-- ============================================================================ +-- NOTHING is the spelling for a control that is deliberately wired to no +-- action: a button that only carries a caption for a layout, a container that +-- is styled as a card but is not clickable. It writes Forms$NoAction. +-- +-- It is here because it only became a real action expression in +-- mendixlabs/mxcli#1062. Before that it was documented and used but absent from +-- the grammar: it reached Forms$NoAction by FAILING to parse as an action and +-- falling through to the generic property branch, which stores the slot as a +-- plain string. `Action: OPEN_LINK` (a real keyword short its argument) and +-- `Action: TOTALLY_MADE_UP` took the same route to the same dead widget, with +-- check, exec and mxbuild all clean — so the fall-through could not be rejected +-- until this form had a branch of its own. +-- +-- The gate is the point: this must survive `exec` into a real project and +-- `mx check`, which is what distinguishes a promoted grammar form from one that +-- parses and then cannot be written (the builder's action switch ends in a +-- `default:` that refuses unknown types). +create page PgTest.P014_InertControls ( + Title: 'Inert controls', + Layout: Atlas_Core.Atlas_Default +) +{ + actionbutton btnDecorative (Caption: 'Decorative', Action: NOTHING) + linkbutton lnkDecorative (Caption: 'Also decorative', action: nothing) + container cCard (OnClick: NOTHING, Class: 'card') { + dynamictext txtCard (Content: 'A card that is not clickable') + } +} diff --git a/mdl/executor/cmd_fragments.go b/mdl/executor/cmd_fragments.go index cf005b5e7..15b5782f0 100644 --- a/mdl/executor/cmd_fragments.go +++ b/mdl/executor/cmd_fragments.go @@ -260,6 +260,8 @@ func formatDataSourceV3(ds *ast.DataSourceV3) string { func formatActionV3(a *ast.ActionV3) string { switch a.Type { + case "none": + return "nothing" case "save": if a.ClosePage { return "save_changes close_page" diff --git a/mdl/executor/cmd_pages_builder_action_nothing_test.go b/mdl/executor/cmd_pages_builder_action_nothing_test.go new file mode 100644 index 000000000..22de885b4 --- /dev/null +++ b/mdl/executor/cmd_pages_builder_action_nothing_test.go @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// Promoting NOTHING to a real actionExprV3 alternative changes what reaches the +// builder: the slot used to hold a string the builder never saw, and now holds +// an *ast.ActionV3 with Type "none". buildClientActionV3's switch ends in a +// `default:` that returns "unsupported action type", so without the matching +// case this change would turn a documented, working spelling into an exec +// failure — the grammar and the builder have to move together +// (mendixlabs/mxcli#1062). +// +// The resulting element is the same Forms$NoAction the fall-through produced, so +// nothing changes on disk. Measured end to end: a page written by the pre-fix +// binary and re-run by the post-fix binary reports `Unchanged page`, which is +// the repo's own canonical comparison (ADR-0008) saying the documents are +// semantically identical. +func TestBuildClientAction_NothingIsNoAction(t *testing.T) { + pb := &pageBuilder{} + + got, err := pb.buildClientActionV3(&ast.ActionV3{Type: "none"}) + if err != nil { + t.Fatalf("buildClientActionV3(none): %v — `Action: NOTHING` is shipped syntax", err) + } + no, ok := got.(*pages.NoClientAction) + if !ok { + t.Fatalf("action type = %T, want *pages.NoClientAction", got) + } + if no.TypeName != "Forms$NoAction" { + t.Errorf("TypeName = %q, want Forms$NoAction", no.TypeName) + } + if no.ID == "" { + t.Error("no element ID minted") + } +} + +// The control: an action type the builder does not know must still be refused, +// or the case above would have been better written as a silent default. +func TestBuildClientAction_UnknownTypeIsStillRefused(t *testing.T) { + pb := &pageBuilder{} + if _, err := pb.buildClientActionV3(&ast.ActionV3{Type: "totallyMadeUp"}); err == nil { + t.Fatal("an unknown action type was accepted — the default branch is what keeps a " + + "new grammar form from being written as nothing") + } +} diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index 71dcee077..b4ed469a0 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -1283,6 +1283,20 @@ func (pb *pageBuilder) getNanoflowReturnEntityName(qualifiedName string) string // buildClientActionV3 converts a V3 Action AST to a pages.ClientAction. func (pb *pageBuilder) buildClientActionV3(action *ast.ActionV3) (pages.ClientAction, error) { switch action.Type { + case "none": + // `Action: NOTHING` — deliberately inert. The same Forms$NoAction the + // default branch of serializeClientAction has always produced for this + // spelling; what is new is that it arrives as an action rather than as a + // string the grammar failed to parse. Without this case the promotion in + // actionExprV3 would turn a documented, working spelling into + // "unsupported action type" at exec (mendixlabs/mxcli#1062). + return &pages.NoClientAction{ + BaseElement: model.BaseElement{ + ID: model.ID(types.GenerateID()), + TypeName: "Forms$NoAction", + }, + }, nil + case "save": return &pages.SaveChangesClientAction{ BaseElement: model.BaseElement{ diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index c27671d1a..e498d12df 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -561,8 +561,22 @@ associationPathV3 ; // V3 Action expressions +// +// NOTHING is a real alternative, not a courtesy. It is the documented spelling +// for a deliberately inert button (docs-site/src/language/alter-page.md, the +// quick reference, the synced alter-page skill, nine mdl-examples scripts) and +// it was never in this rule: it reached Forms$NoAction by FAILING to match here +// and falling through to `keyword COLON propertyValueV3` at the end of +// widgetPropertyV3, which stores the slot as a plain string. +// +// That fall-through is what mendixlabs/mxcli#1062 reports: `Action: OPEN_LINK` +// (a real keyword short its argument) and `Action: TOTALLY_MADE_UP` take the +// same route to the same NoAction, silently. The scalar cannot be rejected +// while the documented form still depends on it, so the promotion below is the +// half of the fix that makes MDL-WIDGET28 possible. actionExprV3 : VARIABLE // $handler — a fragment action parameter (see fragmentParam) + | NOTHING // NOTHING — an explicitly inert widget (Forms$NoAction) | SAVE_CHANGES (CLOSE_PAGE)? // SAVE_CHANGES or SAVE_CHANGES CLOSE_PAGE | CANCEL_CHANGES (CLOSE_PAGE)? // CANCEL_CHANGES | CLOSE_PAGE // CLOSE_PAGE diff --git a/mdl/visitor/visitor_page_action_nothing_test.go b/mdl/visitor/visitor_page_action_nothing_test.go new file mode 100644 index 000000000..f5c96c924 --- /dev/null +++ b/mdl/visitor/visitor_page_action_nothing_test.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func actionSlotValue(t *testing.T, src, key string) any { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parsing %q: %v", src, errs) + } + st, ok := prog.Statements[0].(*ast.CreatePageStmtV3) + if !ok { + t.Fatalf("statement type = %T, want *ast.CreatePageStmtV3", prog.Statements[0]) + } + return st.Widgets[0].Properties[key] +} + +// `NOTHING` is the documented spelling for a deliberately inert widget and was +// never an actionExprV3 alternative: it reached Forms$NoAction by FAILING to +// match the action rule and falling through to `keyword COLON propertyValueV3`, +// which stores the slot as a plain string. +// +// That is why mendixlabs/mxcli#1062 could not simply be rejected — the same +// fall-through carries `Action: OPEN_LINK` and `Action: TOTALLY_MADE_UP`, so +// MDL-WIDGET28 would have flagged working, shipped syntax. Promoting NOTHING is +// what separates them. +func TestAction_NothingParsesAsAnAction(t *testing.T) { + for _, tt := range []struct{ name, src, key string }{ + {"Action upper", "create page M.P (Title: 'x', Layout: A.L) { actionbutton b (Action: NOTHING) };", "Action"}, + {"action lower", "create page M.P (Title: 'x', Layout: A.L) { actionbutton b (action: nothing) };", "Action"}, + {"OnClick", "create page M.P (Title: 'x', Layout: A.L) { container c (OnClick: NOTHING) { dynamictext t (Content: 'x') } };", "Action"}, + {"OnChange", "create page M.P (Title: 'x', Layout: A.L) { textbox tb (Attribute: Name, OnChange: NOTHING) };", "OnChange"}, + } { + t.Run(tt.name, func(t *testing.T) { + raw := actionSlotValue(t, tt.src, tt.key) + action, ok := raw.(*ast.ActionV3) + if !ok { + t.Fatalf("%s = %T (%v), want *ast.ActionV3 — a scalar here is the fall-through "+ + "MDL-WIDGET28 rejects, and would make this documented spelling an error", + tt.key, raw, raw) + } + if action.Type != "none" { + t.Errorf("Type = %q, want none", action.Type) + } + }) + } +} + +// The fall-through itself, pinned. MDL-WIDGET28 detects the fault by finding a +// non-action in the slot, so if the grammar ever started REJECTING these at +// parse time the rule would go quiet and this test would say why. +func TestAction_UnmatchedExpressionFallsThroughToAScalar(t *testing.T) { + for _, value := range []string{"OPEN_LINK", "SHOW_PAGE", "CREATE_OBJECT", "COMPLETE_TASK", "TOTALLY_MADE_UP"} { + src := "create page M.P (Title: 'x', Layout: A.L) { actionbutton b (Action: " + value + ") };" + raw := actionSlotValue(t, src, "Action") + if _, isAction := raw.(*ast.ActionV3); isAction { + t.Errorf("Action: %s parsed as an action — if the grammar now matches it, "+ + "MDL-WIDGET28 needs revisiting rather than this test", value) + } + if got, ok := raw.(string); !ok || got != value { + t.Errorf("Action: %s stored as %T %v, want the string %q", value, raw, raw, value) + } + } +} diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index 798497a78..b5f16e5fa 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -1016,6 +1016,13 @@ func buildActionV3(ctx parser.IActionExprV3Context) *ast.ActionV3 { // $handler — a fragment action parameter; resolved at expansion. action.Type = "param" action.Target = strings.TrimPrefix(v.GetText(), "$") + } else if actCtx.NOTHING() != nil { + // An explicitly inert widget. Byte-identical to what the scalar + // fall-through already produced (Forms$NoAction) — what changes is that + // the slot now holds an *ast.ActionV3, so a scalar left in it means the + // action expression failed to parse rather than "the author wrote + // NOTHING". See MDL-WIDGET28 (mendixlabs/mxcli#1062). + action.Type = "none" } else if actCtx.SAVE_CHANGES() != nil { action.Type = "save" action.ClosePage = actCtx.CLOSE_PAGE() != nil From 56f95e75b1a786fded3813eda731da2ce2621ff3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 05:32:34 +0000 Subject: [PATCH 08/18] Report a non-action in a widget's action slot (MDL-WIDGET28) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mendixlabs/mxcli#1062: `Action: OPEN_LINK` (a real keyword short its argument) and `Action: TOTALLY_MADE_UP` (a token that was never a keyword) both produced Forms$NoAction — a widget that rendered, carried its caption, and did nothing. Measured on Mendix 11.12.0, three pages in one script: `mxcli check` passed, `exec` reported "Created page", `mx check` reported 0 errors, and `describe page` came back with no action on the widget at all, the token appearing nowhere in the unit. Nothing downstream could catch it, because a no-action widget is legal Mendix — only mxcli saw the author ask for an action. `Action:`, `OnClick:` and `OnChange:` each carry actionExprV3, and when the value does not match it ANTLR falls through to the generic property branch rather than failing. Any non-*ast.ActionV3 left in one of those slots therefore means the action expression did not parse, and is now an error at check time — which also refuses it at exec before anything is written. `OnClick:` is looked for under its own key as well as "Action": the alias only collapses onto "Action" once it has parsed, so the broken spelling keeps its own. An error rather than a warning, unlike the neighbouring "silently dropped" rules (MDL-WIDGET20/21/23): those fire on values that are valid MDL the writer happens not to route, while this one fires on text that failed to parse as the thing the slot accepts. ALTER PAGE already refused the same value ("Action value must be an action expression") — CREATE PAGE was the inconsistent one. The rule only became possible once NOTHING was a real action expression (the previous commit); before that it would have flagged documented, shipped syntax. Swept every action value in mdl-examples first: `nothing` was the only non-grammar scalar in use, 9 occurrences, and `make check-mdl` passes over the whole corpus. Controls: stubbing validateWidgetActionSlot fails 7 of the new tests with the reported symptom (0 violations where the fault is), and the NOTHING cases go on passing, which is what separates the two halves. Third appearance of this class — SIGN_OUT and OPEN_LINK both used to reach Forms$NoAction through the writer's default branch (CapTrackV2 FINDINGS §10); this is the first at the parser layer. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JEZmExJUvn2nWTWE9mrd4i --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../mendix/create-page/reference/widgets.md | 11 + CHANGELOG.md | 10 + cmd/mxcli/syntax/features_page.go | 5 +- docs-site/src/reference/page/create-page.md | 12 + docs/01-project/MDL_QUICK_REFERENCE.md | 3 +- .../action-slot-not-an-action-1062.fail.mdl | 84 +++++++ mdl/executor/validate_widget_action_slot.go | 131 ++++++++++ .../validate_widget_action_slot_test.go | 230 ++++++++++++++++++ mdl/executor/validate_widgets.go | 5 + 10 files changed, 489 insertions(+), 3 deletions(-) create mode 100644 mdl-examples/bug-tests/action-slot-not-an-action-1062.fail.mdl create mode 100644 mdl/executor/validate_widget_action_slot.go create mode 100644 mdl/executor/validate_widget_action_slot_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 8804d89a7..780ff67c0 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -586,3 +586,4 @@ {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A SOAP `call web service` naming a mapping and service that REALLY EXIST produced a project Mendix could not LOAD: `mx check` stopped before validation with `Mendix.Modeler.Storage.StorageLoadException \u2026 The text 'c2d1682f-09de-4cc7-95a3-82d5ee5ef243' is not a valid ImportMappingIdentifier`. With the load fixed, the same call was still invalid: CE0386 'Operation GetOrder does not exist in consumed web service Clients.OrderSoapClient' (the operation does exist)", "cause": "Two independent defects, both about a name. (1) `resolveMappingRefForWrite` converted the receive mapping's qualified name to the mapping unit's `$ID`; ImportMappingCall.ReturnValueMapping is an ImportMappingIdentifier \u2014 a qualified name \u2014 so a UUID there is unloadable, not merely invalid. (2) ServiceName was derived as the local part of the imported service's qualified name ('OrderSoapClient'), but it is the WSDL `` ('OrdersWS'); Mendix resolves the operation WITHIN the named service, so a wrong one hides every operation", "file": "`mdl/executor/cmd_microflows_builder_calls.go`, `mdl/executor/webservice_names.go` (new)", "insight": "**A green gate can be green BECAUSE the fixture is broken.** The UUID substitution only happened when the mapping lookup SUCCEEDED. The only SOAP fixture (mdl-examples/doctype-tests/06b-soap-examples.mdl) names a service and mappings that do not exist \u2014 deliberately, to demonstrate dangling references \u2014 so exec took the fallback every time and the qualified name survived. A VALID reference was the single input that triggered an unloadable project, and no test used one. When a code path branches on 'did the lookup resolve', the fixture must cover BOTH branches; a fixture built to show error handling covers only one. **Second: fixing one name unmasks the next error, so work the chain against a real project rather than declaring victory at the first green.** Measured on ako/TestApp (11.14.0, baseline 0 errors), the same one-statement script went StorageLoadException -> CE0386 -> CE0243+CE0366+CE0178 as ReturnValueMapping, then ServiceName were fixed; each error was hidden by the one before it. **Third, two API traps found by debugging rather than reading**: `ListRawUnitsByType` matches the $Type EXACTLY despite its parameter being named typePrefix ('WebServices$ImportedService' returns 0, 'WebServices$ImportedServiceImpl' returns 1) \u2014 which is why the pre-existing resolveWebServiceReference, asking for 'WebServices$ImportedWebService', resolves nothing on any real project; and a unit unmarshalled into map[string]any nests sub-documents as maps, not bson.D, so a lookup asserting bson.D finds nothing, which is indistinguishable from 'no such service' and falls back to the wrong answer instead of failing", "refs": []} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A SOAP `call web service` assigning its result was rejected twice over: `[CE0243] \"The mapping used to return a value of type 'Nothing', but now returns a value of type 'Clients.Order'\"` and `[CE0366] \"Cannot store in variable when there is no return value\"` \u2014 on a call whose receive mapping plainly produces an entity", "cause": "Both engines wrote the result handling's VariableType as `DataTypes$VoidType` unconditionally. Void means the call returns nothing, so it contradicts the mapping AND makes the assignment illegal. Studio Pro writes the entity the mapping produces: `DataTypes$ObjectType{Entity: \"Clients.Order\"}`, which is the Entity of the import mapping's ROOT `ImportMappings$ObjectMappingElement`", "file": "`mdl/executor/webservice_names.go` (resolveImportMappingEntity), `sdk/mpr/writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**Fixing one wrong name in a SOAP call reveals the next; work the chain against a real project instead of stopping at the first green.** On ako/TestApp (11.14.0, baseline 0 errors) the SAME one-statement script went StorageLoadException (ReturnValueMapping written as a UUID) -> CE0386 (ServiceName derived instead of read) -> CE0243+CE0366 (VariableType Void) -> CE0178 (operation arguments), four rounds, each error invisible until the previous fix landed. Any of them could have been called 'the' bug. **The enabler each time was reading the referenced DOCUMENT rather than deriving from the statement**: the WSDL service name is in `Description.Services[].Name` of the imported service, the result entity in `Elements[0].Entity` of the import mapping \u2014 both structured, neither needing the embedded WSDL to be parsed. `ListRawUnitsByType` reaches them on both engines with no new backend method, but note it matches the $Type EXACTLY despite the parameter being named typePrefix, and the types are `WebServices$ImportedServiceImpl` and `ImportMappings$ImportMapping` (NOT the `Mappings$` prefix their child elements use). **Every resolver returns \"\" rather than guessing** \u2014 unresolvable, ambiguous, or wrong-shaped all fall back to what shipped, because a made-up name reproduces the same error with different text in it and is harder to recognise. Remaining and measured: CE0178 needs operation arguments, which MDL cannot express at all (callWebServiceStatement has no argument list), and Range.SingleObject differs from Studio Pro with no error yet attached \u2014 the reference mapping roots carry MaxOccurs 1 while the calls carry SingleObject false, so it is not the mapping's cardinality and would be a guess", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "A DESCRIBE-side resolver for SOAP references had been unreachable since it was written, and its unit test passed. `describe microflow` printed the right service and mapping names throughout", "cause": "`resolveWebServiceReference` looked the service up by ELEMENT ID among units of type `WebServices$ImportedWebService`. It could not match on two independent counts: the stored $Type is `WebServices$ImportedServiceImpl` (ImportedWebService is the SDK name, and nothing is stored under it), and the value compared was never an id \u2014 ImportedService is a BY_NAME_REFERENCE, so it already held `Clients.OrderSoapClient`. Same for the two mapping resolvers. Every call fell through to a fallback that returned the stored string, which is the correct answer", "file": "`mdl/executor/cmd_microflows_format_action.go` (formatWebServiceCallAction; the five resolvers removed)", "insight": "**A resolver whose fallback is the correct answer is indistinguishable in its output from one that works \u2014 so only the input side can prove it runs.** Nothing in the DESCRIBE text could ever have been wrong, which is why this survived: the test that covered it, `TestFormatAction_WebServiceCallResolvesKnownReferences`, built a world where ServiceID WAS a unit id and the unit type WAS `ImportedWebService`, neither of which occurs in any project \u2014 the green test asserted the fiction, not the code. The replacement inverts it: the mock backend calls `t.Fatal` if it is consulted at all, so the test fails unless no lookup happens. **The wider fact, measured, is that the structured DESCRIBE branch is unreachable for real SOAP calls anyway**: all three of ako/TestApp's carry 15 keys and `webServiceActionRequiresRawBSON` supports 9, and mxcli's own writer emits the same 15, so every SOAP call on either engine describes as `call web service raw ''`. A branch nothing reaches cannot be validated by any amount of passing tests over it. **Check a BY_NAME_REFERENCE before writing a resolver**: `modelsdk/gen/*/refs.go` states the kind (`codec.RefByName` here), and an existing test two packages away was already passing `Mod.Service` as the value", "refs": []} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget action slot given a real keyword short its argument (`Action: OPEN_LINK` with no URL) or an invented one (`Action: TOTALLY_MADE_UP`) was written as Forms$NoAction: a control that rendered, carried its caption, and did nothing. `mxcli check` passed, `exec` said \"Created page\", `mx check` gave 0 errors, `describe page` showed no action at all", "cause": "actionExprV3 spells `OPEN_LINK STRING_LITERAL`, so a bare OPEN_LINK cannot match it — and ANTLR does not fail, it falls through to the generic `keyword COLON propertyValueV3` alternative at the end of widgetPropertyV3. The slot then holds a plain string; WidgetV3.GetAction type-asserts to *ast.ActionV3, gets nil, and the writer's default branch emits Forms$NoAction. Every under-specified form went the same way (SHOW_PAGE, CREATE_OBJECT, COMPLETE_TASK, MICROFLOW, NANOFLOW), in Action:, OnClick: and OnChange: alike", "file": "`mdl/grammar/domains/MDLPage.g4` (actionExprV3, NOTHING promoted), `mdl/visitor/visitor_page_v3.go` (buildActionV3), `mdl/executor/cmd_pages_builder_v3.go` (buildClientActionV3 case \"none\"), `mdl/executor/validate_widget_action_slot.go` (new, MDL-WIDGET28)", "insight": "**Before rejecting a degraded form, check whether anything DOCUMENTED depends on the degradation.** `Action: NOTHING` — the shipped spelling for a deliberately inert widget, in the docs site, MDL_QUICK_REFERENCE, the synced alter-page skill and nine mdl-examples scripts — was never in the grammar: it reached Forms$NoAction through the very fall-through the report is about. The reporter's own proposal (\"make it a parse error\") would therefore have broken working syntax, and a naive `any scalar in an action slot is an error` rule breaks the example corpus on the first run. The measurement that settled it took one command — sweep every action value in mdl-examples and count them (`nothing` was the only non-grammar scalar, 9 uses) — and it should come BEFORE writing the rule, not after `make check-mdl` fails. The fix is then two coupled halves: promote the documented form to a real grammar alternative, and only then report the rest. Two controls prove the coupling: stub the validator and 7 tests fail with the reported symptom; delete the grammar alternative and the VISITOR NO LONGER COMPILES (`actCtx.NOTHING undefined`), which is a harder coupling than any test. Also: the builder's action switch ends in a `default:` that refuses unknown types, so promoting a grammar form without adding its case turns a working spelling into an exec failure — grammar, visitor and builder move together or not at all. Third appearance of this class: SIGN_OUT and OPEN_LINK both used to reach Forms$NoAction through the WRITER's default branch (CapTrackV2 FINDINGS §10); this is the first at the parser layer, and the general shape is that a silent-degrade path with a legal-looking destination is invisible to every downstream check, because the destination really is legal.", "refs": ["mendixlabs/mxcli#1062"]} diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index 7774da193..da574232d 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -103,6 +103,7 @@ describe icon collection Atlas_Core.Atlas_Filled -- every icon + its reference ``` **Action Bindings:** +- `action: nothing` - Deliberately no action (a decorative button, a card that is not clickable) - `action: save_changes` - Save changes to object - `action: save_changes close_page` - Save and close page - `action: cancel_changes` - Cancel changes @@ -128,6 +129,16 @@ describe icon collection Atlas_Core.Atlas_Filled -- every icon + its reference variable is refused as **MDL-PAGEARG01** — it used to be accepted and silently opened the page with the context object anyway. To open a page with something else, call a microflow that shows it. +- **The list above is the whole vocabulary, and a keyword without its argument is + not in it.** `action: open_link` with no URL, `action: show_page` with no page, + `action: microflow` with no name — each is **MDL-WIDGET28**. Until + mendixlabs/mxcli#1062 these were written as a widget with *no action at all*: + it rendered, carried its caption, and did nothing, while `mxcli check`, `exec` + and mxbuild all reported success, because a no-action widget is legal Mendix. + An invented keyword (`action: totally_made_up`) did the same. Use + `action: nothing` when a control really is meant to be inert, so a dead one + always means a mistake. +- The same forms serve `onclick:` (an alias of `action:`) and `onchange:`. **Button Styles:** `default`, `primary`, `success`, `info`, `warning`, `danger`, `inverse` - Case-insensitive (`primary` and `Primary` both work). diff --git a/CHANGELOG.md b/CHANGELOG.md index 5954f7f55..46de0baf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **An action keyword missing its argument silently wrote a dead widget** (mendixlabs/mxcli#1062) — `Action: OPEN_LINK` without a URL, and the invented `Action: TOTALLY_MADE_UP`, both produced `Forms$NoAction`: a button that rendered, carried its caption, and did nothing. Measured on Mendix 11.12.0, three pages in one script — `mxcli check` passed, `exec` reported "Created page", `mx check` reported **0 errors**, and `describe page` came back with no action on the widget at all, the token appearing nowhere in the unit. Nothing downstream could catch it, because a no-action widget is legal Mendix; only mxcli saw the author ask for an action. + + `actionExprV3` spells `OPEN_LINK STRING_LITERAL`, so a bare `OPEN_LINK` cannot match it — and ANTLR does not fail, it falls through to the generic `keyword COLON propertyValueV3` alternative at the end of `widgetPropertyV3`. The slot then holds a plain string, `GetAction()` type-asserts and returns nil, and the writer emits NoAction. Every under-specified form went the same way, not just the reported one: `SHOW_PAGE`, `CREATE_OBJECT`, `COMPLETE_TASK`, `MICROFLOW` and `NANOFLOW` short their argument, in `Action:`, `OnClick:` and `OnChange:` alike. + + The obvious fix — reject the scalar — was not available, and that is the substance of this change. **`Action: NOTHING` was not in the grammar either.** The documented spelling for a deliberately inert widget (the docs site, `MDL_QUICK_REFERENCE.md`, the synced alter-page skill, nine `mdl-examples` scripts) reached its `Forms$NoAction` through this very fall-through, so rejecting the scalar would have made shipped, working syntax an error. `NOTHING` is now a real `actionExprV3` alternative, and only then is everything else in an action slot reportable — as **MDL-WIDGET28**, an error, which also refuses it at `exec` before anything is written. + + Promoting `NOTHING` changes nothing on disk: it maps to the same `pages.NoClientAction` the default branch already produced, and a page written by the pre-fix binary is reported `Unchanged page` when the post-fix binary re-runs the same script — the repo's own canonical comparison (ADR-0008) saying the documents are semantically identical. The builder's action switch ends in a `default:` that refuses unknown types, so the grammar and the builder had to move together or the promotion would have turned a working spelling into an exec failure. + + `ALTER PAGE` already refused exactly this value (*"Action value must be an action expression"*); `CREATE PAGE` was the inconsistent one. This is also the third appearance of the class — `SIGN_OUT` and `OPEN_LINK` both used to reach `Forms$NoAction` through the *writer's* default branch (CapTrackV2 FINDINGS §10) — and the first at the parser layer. + - **`marketplace update` and `marketplace install` left the model at CE0066, with nothing said about it** (mendixlabs/mxcli#1085) — a headless module update exited 0, reported *"81 units copied, 45 element identities preserved, 6 role grant(s) restored"*, and `mx check` then gave **CE0066** *"Entity access is out of date. Please update security by clicking the 'Update security' button in the domain model editor"* at the module's domain model. The command's own *"Next, repair what a headless update leaves behind"* block named CE0463 and CE6087 only, so the error read as needing Studio Pro — it was reported as *"no headless fix available"*, which is what breaks an unattended upgrade pipeline. A transplant is the one mxcli write path that does **not** reconcile as it writes. `TransplantModule` copies the incoming module's units in verbatim, and `RestoreRoleGrants` runs its statements one at a time rather than as a program, so the executor's finalize step — the only caller of `ReconcileMemberAccesses` outside the entity, grant and association handlers — never runs. An access rule that does not cover every member of its entity therefore arrived exactly as the package shipped it. diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 3b59eebac..a0ee557f7 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -210,14 +210,15 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { Register(SyntaxFeature{ Path: "page.action", - Summary: "Widget actions: save, cancel, close, delete, show page, microflow, nanoflow", + Summary: "Widget actions: save, cancel, close, delete, show page, microflow, nanoflow, nothing", Keywords: []string{ "action", "save", "cancel", "close", "delete", "show page", "navigate", "microflow", "create object", "button style", "primary", "danger", "success", "icon", "linkbutton", "link button", + "nothing", "no action", "inert", "dead button", }, - Syntax: "Action: SAVE_CHANGES\nAction: SAVE_CHANGES CLOSE_PAGE -- save, then close the pop-up\nAction: CANCEL_CHANGES\nAction: CANCEL_CHANGES CLOSE_PAGE\nAction: CLOSE_PAGE\nAction: DELETE\nAction: DELETE CLOSE_PAGE\nAction: DELETE_OBJECT\nAction: NANOFLOW Module.NF\nAction: NANOFLOW Module.NF(Param: $val)\nAction: OPEN_LINK 'https://example.com'\nAction: SIGN_OUT\nAction: COMPLETE_TASK 'OutcomeName'\nAction: SHOW_PAGE Module.Page\nAction: SHOW_PAGE Module.Page(Param: $currentObject)\nAction: MICROFLOW Module.MF\nAction: MICROFLOW Module.MF(Param: $val)\nAction: CREATE_OBJECT Module.Entity THEN SHOW_PAGE Module.Page\n\nA microflow or nanoflow action is a CALL: it needs an argument for every\nparameter the flow declares, or Mendix rejects the page with CE1571. The\nargument list is the same on every widget that takes an action -- a\nCONTAINER (which is clickable) as much as an ACTIONBUTTON. An enclosing\ndata container of the right type supplies it without an argument; a data\ngrid's CONTROL BAR does not, because it is not row-scoped -- pass the\ngrid's selection there (`$dgOrders`).\n\nA SHOW_PAGE argument must be the enclosing widget's context object --\neither $currentObject or the name of the variable the enclosing data\nwidget is bound to. Mendix infers it from that widget, so naming any\nother variable is refused (MDL-PAGEARG01); call a microflow instead.\n\nOPEN_LINK takes a static web address and stores it as a\nForms$StaticOrDynamicString. Mendix also supports a DYNAMIC address, read\nfrom an attribute at runtime; MDL cannot author that one, and DESCRIBE\nflags such a button rather than printing its address as a literal.\n\nButton styles: Default, Primary, Success, Info, Warning, Danger\nIcon: 'Module.IconCollection.IconName' -- e.g. 'Atlas_Core.Atlas_Filled.pencil'\nUse `linkbutton` instead of `actionbutton` for link render mode (same properties).", + Syntax: "Action: NOTHING -- deliberately no action (Forms$NoAction)\nAction: SAVE_CHANGES\nAction: SAVE_CHANGES CLOSE_PAGE -- save, then close the pop-up\nAction: CANCEL_CHANGES\nAction: CANCEL_CHANGES CLOSE_PAGE\nAction: CLOSE_PAGE\nAction: DELETE\nAction: DELETE CLOSE_PAGE\nAction: DELETE_OBJECT\nAction: NANOFLOW Module.NF\nAction: NANOFLOW Module.NF(Param: $val)\nAction: OPEN_LINK 'https://example.com'\nAction: SIGN_OUT\nAction: COMPLETE_TASK 'OutcomeName'\nAction: SHOW_PAGE Module.Page\nAction: SHOW_PAGE Module.Page(Param: $currentObject)\nAction: MICROFLOW Module.MF\nAction: MICROFLOW Module.MF(Param: $val)\nAction: CREATE_OBJECT Module.Entity THEN SHOW_PAGE Module.Page\n\nThe list above is exhaustive. Anything else in an action slot is an\nERROR (MDL-WIDGET28), including a real keyword short its argument --\n`Action: OPEN_LINK` without a URL, `Action: SHOW_PAGE` without a page.\nSuch a widget used to be written with NO action at all and rendered as a\ndead control, with check, exec and mxbuild all clean, because a\nno-action widget is legal Mendix (mendixlabs/mxcli#1062). Write NOTHING\nwhen a control really is meant to be inert.\n\nThe same forms serve `OnClick:` (an alias of `Action:`) and `OnChange:`.\n\nA microflow or nanoflow action is a CALL: it needs an argument for every\nparameter the flow declares, or Mendix rejects the page with CE1571. The\nargument list is the same on every widget that takes an action -- a\nCONTAINER (which is clickable) as much as an ACTIONBUTTON. An enclosing\ndata container of the right type supplies it without an argument; a data\ngrid's CONTROL BAR does not, because it is not row-scoped -- pass the\ngrid's selection there (`$dgOrders`).\n\nA SHOW_PAGE argument must be the enclosing widget's context object --\neither $currentObject or the name of the variable the enclosing data\nwidget is bound to. Mendix infers it from that widget, so naming any\nother variable is refused (MDL-PAGEARG01); call a microflow instead.\n\nOPEN_LINK takes a static web address and stores it as a\nForms$StaticOrDynamicString. Mendix also supports a DYNAMIC address, read\nfrom an attribute at runtime; MDL cannot author that one, and DESCRIBE\nflags such a button rather than printing its address as a literal.\n\nButton styles: Default, Primary, Success, Info, Warning, Danger\nIcon: 'Module.IconCollection.IconName' -- e.g. 'Atlas_Core.Atlas_Filled.pencil'\nUse `linkbutton` instead of `actionbutton` for link render mode (same properties).", Example: "ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\nACTIONBUTTON btnEdit (Caption: 'Edit',\n Action: SHOW_PAGE Module.EditPage(Item: $currentObject))\nLINKBUTTON btnDelete (Caption: 'Delete', Action: DELETE,\n Icon: 'Atlas_Core.Atlas_Filled.pencil')\n\n-- A clickable CONTAINER in a data grid's control bar, calling a nanoflow\n-- with the grid's selection as its argument.\nDATAGRID dgOrders (DataSource: DATABASE FROM Sales.Order, Selection: Single) {\n COLUMN colNr (Attribute: Number, Caption: 'Order #')\n CONTROLBAR cb {\n CONTAINER cShip (Class: 'command',\n Action: NANOFLOW Sales.ACT_Ship($Order = $dgOrders)) {\n ACTIONBUTTON btnShip (Caption: 'Ship')\n }\n }\n}", SeeAlso: []string{"page.widgets"}, }) diff --git a/docs-site/src/reference/page/create-page.md b/docs-site/src/reference/page/create-page.md index 459f08153..43766d97b 100644 --- a/docs-site/src/reference/page/create-page.md +++ b/docs-site/src/reference/page/create-page.md @@ -125,6 +125,18 @@ values, including the argument list. | Page | `Action: PAGE Module.PageName` | Opens a page | | Close | `Action: CLOSE_PAGE` | Closes the current page | | Delete | `Action: DELETE` | Deletes the context object | +| Nothing | `Action: NOTHING` | Deliberately no action — a decorative button, a card that is not clickable | + +The set is closed. Anything else in an action slot is an error +(**MDL-WIDGET28**), and that includes a real action keyword **missing its +argument** — `Action: OPEN_LINK` with no URL, `Action: SHOW_PAGE` with no page. +Such a widget used to be written with no action at all: it rendered, carried its +caption, and did nothing, while `mxcli check`, `exec` and mxbuild all reported +success, because a no-action widget is perfectly legal Mendix. Write `NOTHING` +when a control is genuinely meant to be inert, so that a dead one always means a +mistake. + +The same values serve `OnClick:` (an alias of `Action:`) and `OnChange:`. A microflow or nanoflow action is a **call**: every parameter the flow declares needs an argument, or Mendix rejects the page with **CE1571**. An enclosing data diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 2a625246b..89f8ecade 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1332,7 +1332,8 @@ MDL uses explicit property declarations for pages: | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | | Variable binding | `datasource: $Var` | `dataview dv (datasource: $Product) { ... }` | -| Action binding | `action: type` | `actionbutton btn (caption: 'Save', action: save_changes)` | +| Action binding | `action: type` | `actionbutton btn (caption: 'Save', action: save_changes)` — the forms are a closed set (`mxcli syntax page.action`); anything else is **MDL-WIDGET28** | +| No action | `action: nothing` | `actionbutton btn (caption: 'Decorative', action: nothing)` — an explicitly inert control. Write it deliberately: an action keyword **short its argument** (`action: open_link` with no URL) is now an error rather than a widget silently written with no action at all | | Microflow action | `action: microflow Name(Param: val)` | `action: microflow Mod.ACT_Process(Order: $Order)` | | Button icon | `icon: 'Module.IconCollection.IconName'` | `linkbutton btn (caption: 'Edit', action: nothing, icon: 'Atlas_Core.Atlas_Filled.pencil')` — icon-collection icon; MxBuild rejects an unknown name (CE1613) | | Clickable container | `onclick: action` (alias of `action:`) | `container card (onclick: microflow Mod.ACT_Open) { ... }` — takes an argument list like a button: `action: nanoflow Mod.ACT_Ship($Order = $dgOrders)` | diff --git a/mdl-examples/bug-tests/action-slot-not-an-action-1062.fail.mdl b/mdl-examples/bug-tests/action-slot-not-an-action-1062.fail.mdl new file mode 100644 index 000000000..c6a1d0406 --- /dev/null +++ b/mdl-examples/bug-tests/action-slot-not-an-action-1062.fail.mdl @@ -0,0 +1,84 @@ +-- mendixlabs/mxcli#1062 — an action keyword MISSING ITS ARGUMENT silently +-- writes a dead widget. +-- +-- NEGATIVE test (.fail.mdl): this script must FAIL `mxcli check`. It can be a +-- .fail.mdl — unlike the #1082 repro beside it — because MDL-WIDGET28 needs no +-- project: the fault is the SHAPE of the value in the slot, which is settled by +-- the parse alone. `make check-mdl` runs check with no project, so a rule that +-- depended on a stored signature would read as regressed here (#891, #892). +-- +-- REPORTED SYMPTOM +-- +-- mxcli 0.21.0, Mendix 11.11.0 / 11.12.x. Three spellings, three exit-0 runs: +-- +-- Action: OPEN_LINK 'https://example.com' → Forms$OpenLinkClientAction +-- Action: OPEN_LINK → Forms$NoAction, check passed +-- Action: TOTALLY_MADE_UP → Forms$NoAction, check passed +-- +-- "an unrecognized or under-specified token should be a parse error rather +-- than a silent degrade to NoAction." +-- +-- WHY IT HAPPENED +-- +-- `actionExprV3` spells `OPEN_LINK STRING_LITERAL`, so a bare OPEN_LINK +-- cannot match it — and ANTLR does not fail, it falls through to the generic +-- `keyword COLON propertyValueV3` alternative at the end of widgetPropertyV3. +-- The slot ends up holding a plain string, GetAction() type-asserts and +-- returns nil, and the writer emits Forms$NoAction. +-- +-- Measured on a Mendix 11.12.0 app: `mxcli check` passed, `exec` reported +-- "Created page", `mx check` reported 0 errors, and `describe page` came back +-- with no action on the button at all. The token appears nowhere in the unit. +-- Nothing downstream can catch it, because a no-action button is legal +-- Mendix — which is exactly why mxcli has to, having seen the author ask for +-- an action. +-- +-- Every under-specified form degraded the same way, not just OPEN_LINK: +-- SHOW_PAGE, CREATE_OBJECT, COMPLETE_TASK, MICROFLOW and NANOFLOW without +-- their argument, in `Action:`, `OnClick:` and `OnChange:` alike. +-- +-- WHY THE OBVIOUS FIX WAS NOT AVAILABLE +-- +-- `Action: NOTHING` — the documented spelling for a deliberately inert widget +-- (docs-site, MDL_QUICK_REFERENCE, the synced alter-page skill, nine scripts +-- in mdl-examples) — was NOT in the grammar either, and reached its +-- Forms$NoAction through this same fall-through. Rejecting the scalar would +-- have made shipped, working syntax an error. So NOTHING was promoted to a +-- real actionExprV3 alternative first; see the positive case in +-- doctype-tests/03-page-examples.mdl (PgTest.P014_InertControls). +-- +-- Every widget below is a violation. The corrected spelling is in the comment +-- beside each one. + +create or replace page BugTest1062.DeadControls +( + title: 'Dead controls', + layout: Atlas_Core.Atlas_Default +) +{ + -- The reported case: a real keyword short its argument. + -- Correct: Action: OPEN_LINK 'https://example.com' + actionbutton btnLink (caption: 'Open the site', Action: OPEN_LINK) + + -- The reported case: a token that was never a keyword. + -- Correct: Action: NOTHING, if the button really is decorative. + actionbutton btnInvented (caption: 'Do the thing', Action: TOTALLY_MADE_UP) + + -- The same fall-through on the other under-specified keywords. + -- Correct: Action: SHOW_PAGE BugTest1062.SomePage + actionbutton btnShow (caption: 'Details', Action: SHOW_PAGE) + + -- Correct: Action: MICROFLOW BugTest1062.ACT_Something + actionbutton btnFlow (caption: 'Run', Action: MICROFLOW) + + -- Correct: Action: COMPLETE_TASK 'Approve' + actionbutton btnTask (caption: 'Approve', Action: COMPLETE_TASK) + + -- `OnClick:` is an alias for `Action:` and degrades identically. It keeps its + -- own property key when it does NOT parse, so a rule looking only at "Action" + -- would miss exactly the broken spelling. + -- Correct: OnClick: MICROFLOW BugTest1062.ACT_Something + container cCard (OnClick: TOTALLY_MADE_UP) { + dynamictext txt (content: 'A card that does nothing') + } +} diff --git a/mdl/executor/validate_widget_action_slot.go b/mdl/executor/validate_widget_action_slot.go new file mode 100644 index 000000000..f4c2a581a --- /dev/null +++ b/mdl/executor/validate_widget_action_slot.go @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time validation for a widget's action slot holding something that is +// not an action. +// +// `Action:`, `OnClick:` and `OnChange:` each have a dedicated grammar branch +// carrying actionExprV3. When the value does not match that rule — a real +// keyword short its argument (`Action: OPEN_LINK`), or a token that was never a +// keyword (`Action: TOTALLY_MADE_UP`) — ANTLR does not fail: it falls through to +// the generic `keyword COLON propertyValueV3` alternative at the end of +// widgetPropertyV3 and the slot ends up holding a plain string. WidgetV3's +// GetAction/GetOnChange type-assert to *ast.ActionV3, get nil, and the widget is +// written with Forms$NoAction. +// +// Nothing downstream notices. A no-action button is legal Mendix, so the build +// is clean — measured on 11.12.0: `mxcli check` passed, `exec` reported "Created +// page", `mx check` reported 0 errors, and `describe page` came back with no +// action on the widget at all. The author gets a button that renders, says +// "Unlink", and does nothing (mendixlabs/mxcli#1062). +// +// This is the third appearance of the class. SIGN_OUT and OPEN_LINK both used to +// reach Forms$NoAction through the *writer's* default branch (CapTrackV2 +// FINDINGS §10, cited in sdk/mpr/writer_widgets_action.go); this is the same +// silent degrade one layer up, in the parser. +// +// The rule is only possible because `NOTHING` was promoted to a real +// actionExprV3 alternative in the same change. It is the documented spelling for +// a deliberately inert widget and it reached Forms$NoAction by exactly this +// fall-through, so before the promotion "scalar in an action slot" covered the +// working case and the two broken ones alike. +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// actionSlotKeys are the property keys with a dedicated actionExprV3 branch. +// +// `Action:` and `OnClick:` are aliases that both land on Properties["Action"] +// once parsed (issue #603), but an *unparsed* `OnClick:` keeps its own key — +// the generic fall-through stores the property under the name as written — so +// both spellings have to be looked for here. +// +// A pluggable widget's NAMED action slot (`createFileAction: …`) degrades the +// same way and is deliberately not listed: for an arbitrary widget property a +// string is an ordinary value, so the fault is only decidable against a loaded +// widget definition, which `mxcli check` has no project to supply. +var actionSlotKeys = []string{"Action", "OnClick", "OnChange"} + +// underSpecified maps an action keyword to what it is missing, so the message +// can name the one token the author left out rather than printing the whole +// grammar. Keyed lowercase; looked up case-insensitively. +var underSpecified = map[string]string{ + "open_link": "a URL — `Action: OPEN_LINK 'https://example.com'`", + "complete_task": "an outcome name — `Action: COMPLETE_TASK 'Approved'`", + "show_page": "a page — `Action: SHOW_PAGE Module.Page`", + "create_object": "an entity — `Action: CREATE_OBJECT Module.Entity`", + "microflow": "a microflow — `Action: MICROFLOW Module.Flow`", + "nanoflow": "a nanoflow — `Action: NANOFLOW Module.Flow`", +} + +// validateWidgetActionSlot reports (MDL-WIDGET28) an action slot whose value is +// not an action expression. +// +// An error, not a warning, unlike the neighbouring "silently dropped" rules +// (MDL-WIDGET20/21/23). Those fire on values that are *valid MDL* the writer +// happens not to route; this one fires on text that failed to parse as the thing +// the slot accepts, and after NOTHING was promoted there is no scalar spelling +// of an action left for it to catch by mistake. ALTER PAGE already refuses the +// same value ("Action value must be an action expression") — CREATE PAGE was the +// inconsistent one. +func validateWidgetActionSlot(w *ast.WidgetV3, locationPrefix string) []linter.Violation { + if w == nil { + return nil + } + var out []linter.Violation + // Iterated over a fixed slice rather than over w.Properties: map order is + // not stable, and a widget with two faulty slots must report them the same + // way every run (CLAUDE.md, determinism). + for _, key := range actionSlotKeys { + raw, present := w.Properties[key] + if !present { + continue + } + if _, ok := raw.(*ast.ActionV3); ok { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL-WIDGET28", + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s: widget `%s` has `%s: %s`, which is not an action — the widget is written with "+ + "no action at all and renders as a dead control (Mendix accepts it, so the build stays clean)", + locationPrefix, w.Name, key, renderActionSlotValue(raw)), + Suggestion: actionSlotSuggestion(raw), + }) + } + return out +} + +// renderActionSlotValue prints the offending value the way the author wrote it, +// so the message can be matched against the source line. +func renderActionSlotValue(raw any) string { + if s, ok := raw.(string); ok { + return s + } + return fmt.Sprintf("%v", raw) +} + +// actionSlotSuggestion names the missing token when the value is a real action +// keyword, and otherwise lists what the slot takes. +// +// The split matters: an author who wrote `OPEN_LINK` knows which action they +// want and needs one argument, while an author who wrote `TOTALLY_MADE_UP` +// needs the vocabulary. Telling the first one to "see `mxcli syntax +// page.action`" buries the answer they were one token away from. +func actionSlotSuggestion(raw any) string { + if s, ok := raw.(string); ok { + if missing, known := underSpecified[strings.ToLower(strings.TrimSpace(s))]; known { + return fmt.Sprintf("`%s` is a real action but takes %s.", s, missing) + } + } + return "Use an action expression — SAVE_CHANGES, CANCEL_CHANGES, CLOSE_PAGE, DELETE_OBJECT, " + + "SIGN_OUT, SHOW_PAGE Module.Page, MICROFLOW Module.Flow, NANOFLOW Module.Flow, " + + "OPEN_LINK 'url', COMPLETE_TASK 'Outcome', CREATE_OBJECT Module.Entity — " + + "or `NOTHING` if the widget is meant to do nothing. See `mxcli syntax page.action`." +} diff --git a/mdl/executor/validate_widget_action_slot_test.go b/mdl/executor/validate_widget_action_slot_test.go new file mode 100644 index 000000000..fad28a94c --- /dev/null +++ b/mdl/executor/validate_widget_action_slot_test.go @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// widget28 parses a script and returns only the MDL-WIDGET28 violations. +// +// It goes through visitor.Build rather than hand-built WidgetV3 literals +// BECAUSE the rule is about what the PARSER does with an unmatched action +// expression. A hand-built `Properties{"Action": "OPEN_LINK"}` would assert the +// same string comparison while proving nothing about whether the grammar still +// produces it — and the grammar is half the fix (mendixlabs/mxcli#1062). +func widget28(t *testing.T, src string) []linter.Violation { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parsing %q: %v", src, errs) + } + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("built-in widget registry not available") + } + var out []linter.Violation + for _, stmt := range prog.Statements { + for _, v := range ValidateWidgetPropertiesForStatement(stmt, registry) { + if v.RuleID == "MDL-WIDGET28" { + out = append(out, v) + } + } + } + return out +} + +func page28(body string) string { + return fmt.Sprintf("create page M.P (Title: 'x', Layout: Atlas_Core.Atlas_Default) {\n %s\n}", body) +} + +// The reported case: a real action keyword short its argument. Measured on +// Mendix 11.12.0 before the fix — `mxcli check` passed, exec said "Created +// page", `mx check` reported 0 errors, and `describe page` came back with no +// action on the button at all. +func TestActionSlot_UnderSpecifiedKeywordIsReported(t *testing.T) { + got := widget28(t, page28(`actionbutton b (Caption: 'Go', Action: OPEN_LINK)`)) + if len(got) != 1 { + t.Fatalf("got %d MDL-WIDGET28, want 1: %+v", len(got), got) + } + if got[0].Severity != linter.SeverityError { + t.Errorf("severity = %v, want error — a warning would let exec write the dead button, "+ + "which is the whole complaint", got[0].Severity) + } + if !strings.Contains(got[0].Message, "OPEN_LINK") || !strings.Contains(got[0].Message, "`b`") { + t.Errorf("message must name the value and the widget:\n%s", got[0].Message) + } + // The author was one token from working syntax; the suggestion has to say + // which token rather than sending them to the grammar. + if !strings.Contains(got[0].Suggestion, "https://example.com") { + t.Errorf("suggestion must show OPEN_LINK's missing argument:\n%s", got[0].Suggestion) + } +} + +// The other half of the report: a token that was never a keyword. +func TestActionSlot_InventedKeywordIsReported(t *testing.T) { + got := widget28(t, page28(`actionbutton b (Caption: 'Go', Action: TOTALLY_MADE_UP)`)) + if len(got) != 1 { + t.Fatalf("got %d MDL-WIDGET28, want 1: %+v", len(got), got) + } + if !strings.Contains(got[0].Suggestion, "NOTHING") { + t.Errorf("an author who invented a keyword needs the vocabulary, including the inert "+ + "spelling:\n%s", got[0].Suggestion) + } +} + +// THE CONTROL FOR THE WHOLE RULE. +// +// `Action: NOTHING` is the documented spelling for a deliberately inert widget +// (docs-site, the quick reference, the synced alter-page skill, nine +// mdl-examples scripts) and it was NOT in actionExprV3 — it reached +// Forms$NoAction through the same fall-through the two tests above report. If +// the grammar promotion is reverted this test fails, which is what pins the two +// halves of the fix together: the scalar cannot be rejected while the working +// spelling still depends on it. +func TestActionSlot_NothingIsARealActionAndStaysClean(t *testing.T) { + for _, body := range []string{ + `actionbutton b (Caption: 'Inert', Action: NOTHING)`, + `actionbutton b (Caption: 'Inert', action: nothing)`, + `container c (OnClick: NOTHING) { dynamictext t (Content: 'x') }`, + } { + if got := widget28(t, page28(body)); len(got) != 0 { + t.Errorf("%s\n reported %d violations, want 0 — this is documented, shipped syntax: %+v", + body, len(got), got) + } + } +} + +// Every real action form must survive the rule. Cheap to assert and it is the +// difference between "rejects scalars" and "rejects everything it does not +// recognise". +func TestActionSlot_RealActionsStayClean(t *testing.T) { + for _, body := range []string{ + `actionbutton b (Caption: 'x', Action: SAVE_CHANGES)`, + `actionbutton b (Caption: 'x', Action: SAVE_CHANGES CLOSE_PAGE)`, + `actionbutton b (Caption: 'x', Action: CANCEL_CHANGES)`, + `actionbutton b (Caption: 'x', Action: CLOSE_PAGE)`, + `actionbutton b (Caption: 'x', Action: DELETE_OBJECT)`, + `actionbutton b (Caption: 'x', Action: SIGN_OUT)`, + `actionbutton b (Caption: 'x', Action: OPEN_LINK 'https://example.com')`, + `actionbutton b (Caption: 'x', Action: COMPLETE_TASK 'Approved')`, + `actionbutton b (Caption: 'x', Action: SHOW_PAGE M.Other)`, + `actionbutton b (Caption: 'x', Action: MICROFLOW M.ACT_Go)`, + `actionbutton b (Caption: 'x', Action: NANOFLOW M.NF_Go)`, + `actionbutton b (Caption: 'x', Action: CREATE_OBJECT M.Thing THEN SHOW_PAGE M.Other)`, + } { + if got := widget28(t, page28(body)); len(got) != 0 { + t.Errorf("%s\n reported %d violations, want 0: %+v", body, len(got), got) + } + } +} + +// Each under-specified keyword gets its own missing token named. Written as a +// table because the map behind it is the kind of list that rots silently. +func TestActionSlot_EachKeywordNamesWhatItIsMissing(t *testing.T) { + for _, tt := range []struct{ value, want string }{ + {"OPEN_LINK", "https://example.com"}, + {"COMPLETE_TASK", "COMPLETE_TASK 'Approved'"}, + {"SHOW_PAGE", "SHOW_PAGE Module.Page"}, + {"CREATE_OBJECT", "CREATE_OBJECT Module.Entity"}, + {"microflow", "MICROFLOW Module.Flow"}, + {"nanoflow", "NANOFLOW Module.Flow"}, + } { + got := widget28(t, page28(fmt.Sprintf(`actionbutton b (Caption: 'x', Action: %s)`, tt.value))) + if len(got) != 1 { + t.Errorf("Action: %s — got %d violations, want 1: %+v", tt.value, len(got), got) + continue + } + if !strings.Contains(got[0].Suggestion, tt.want) { + t.Errorf("Action: %s — suggestion must contain %q:\n%s", tt.value, tt.want, got[0].Suggestion) + } + } +} + +// `OnClick:` and `OnChange:` have their own dedicated grammar branches and so +// degrade the same way. `OnClick:` is an ALIAS that lands on Properties["Action"] +// once parsed (#603) but keeps its own key when it does NOT parse — so a rule +// that only looked at "Action" would miss exactly the broken spelling. +func TestActionSlot_EveryDedicatedSlotIsChecked(t *testing.T) { + for _, tt := range []struct{ name, body string }{ + {"Action", `actionbutton b (Caption: 'x', Action: OPEN_LINK)`}, + {"OnClick", `container c (OnClick: TOTALLY_MADE_UP) { dynamictext t (Content: 'x') }`}, + {"OnChange", `textbox tb (Attribute: Name, OnChange: TOTALLY_MADE_UP)`}, + } { + got := widget28(t, page28(tt.body)) + if len(got) != 1 { + t.Errorf("%s: got %d violations, want 1: %+v", tt.name, len(got), got) + continue + } + if !strings.Contains(got[0].Message, tt.name+":") { + t.Errorf("%s: message must name the slot as written:\n%s", tt.name, got[0].Message) + } + } +} + +// A widget nested below the top level is reached by the same walk. +func TestActionSlot_NestedWidgetIsReported(t *testing.T) { + got := widget28(t, page28( + `container outer { container inner { actionbutton b (Caption: 'x', Action: SHOW_PAGE) } }`)) + if len(got) != 1 { + t.Fatalf("got %d violations, want 1: %+v", len(got), got) + } +} + +// Two faulty slots on one widget must come out in the same order every run. +// w.Properties is a map, so iterating it directly would shuffle (CLAUDE.md). +func TestActionSlot_TwoFaultySlotsReportDeterministically(t *testing.T) { + w := &ast.WidgetV3{Name: "tb", Type: "textbox", Properties: map[string]any{ + "Action": "OPEN_LINK", + "OnChange": "TOTALLY_MADE_UP", + }} + initial := validateWidgetActionSlot(w, "page M.P") + if len(initial) != 2 { + t.Fatalf("got %d violations, want 2: %+v", len(initial), initial) + } + first := violationOrder(initial) + for i := 0; i < 20; i++ { + if got := violationOrder(validateWidgetActionSlot(w, "page M.P")); got != first { + t.Fatalf("order changed between runs: %q then %q", first, got) + } + } +} + +func violationOrder(vs []linter.Violation) string { + var parts []string + for _, v := range vs { + parts = append(parts, v.Message) + } + return strings.Join(parts, "|") +} + +// A non-string scalar reaches the slot too (propertyValueV3 admits numbers and +// booleans), and must not panic or be silently accepted. +func TestActionSlot_NonStringScalarIsReported(t *testing.T) { + for _, raw := range []any{42, true, []any{"a"}} { + w := &ast.WidgetV3{Name: "b", Type: "actionbutton", Properties: map[string]any{"Action": raw}} + got := validateWidgetActionSlot(w, "page M.P") + if len(got) != 1 { + t.Errorf("Action: %v (%T) — got %d violations, want 1", raw, raw, len(got)) + continue + } + if !strings.Contains(got[0].Suggestion, "action expression") { + t.Errorf("Action: %v — a value that is not a keyword needs the vocabulary:\n%s", + raw, got[0].Suggestion) + } + } +} + +// A widget with no action slot at all is the commonest widget there is. +func TestActionSlot_NoActionPropertyIsSilent(t *testing.T) { + w := &ast.WidgetV3{Name: "t", Type: "dynamictext", Properties: map[string]any{"Content": "x"}} + if got := validateWidgetActionSlot(w, "page M.P"); len(got) != 0 { + t.Errorf("got %d violations on a widget with no action slot: %+v", len(got), got) + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index edbead1ba..a86fbdb19 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -130,6 +130,11 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // every widget kind and needs no definition: the SHAPE is wrong whatever // the widget declares. out = append(out, validateObjectEntryProperties(w, registry, locationPrefix)...) + // #1062: an action slot holding something that is not an action, which + // used to check clean, exec clean, build clean and render dead. Runs for + // every widget kind and needs no definition, for the same reason as the + // rule above: the SHAPE of the value is wrong whatever the widget is. + out = append(out, validateWidgetActionSlot(w, locationPrefix)...) // #928: contentparams with no `{N}` placeholder to consume them. if lookupWidgetDef(w, registry) != nil { out = append(out, validatePluggableContentParams(w, locationPrefix)...) From 141da93f3e6dbde0b4991ae1b13dd06ee4903975 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 09:08:01 +0000 Subject: [PATCH 09/18] fix(pages): close the last modelsdk widget gaps, refuse the one that was never writable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five widget constructs sent a user to `MXCLI_ENGINE=legacy`, which is one of the reasons the legacy engine cannot be retired. Measured against sdk/pages the reachable set was five, not the twenty-three a name scan suggests: nineteen widget structs are constructed by nothing, and EntityPathSource / ShowHomePageClientAction are written by NEITHER engine, so their fallback named a path legacy could not take either. None of the five was covered by the doctype gate — no example or skill uses those keywords — which is why the suite ran green while four plain widget keywords failed on the default engine. Four are now written by the modelsdk engine: dropdown (Forms$DropDown), staticimage (Forms$StaticImageViewer), dynamicimage (Forms$ImageViewer), and the nanoflow list/grid data source (Forms$NanoflowSource, built raw because gen binds the nanoflow name directly where Studio Pro nests it in Forms$NanoflowSettings). The fifth was not a gap. `statictext` writes Forms$Text, and Mendix has no such type: BOTH engines produced a project that could not be LOADED — TypeCacheUnknownTypeException, which stops `mx check` and Studio Pro before any validation, so the page is unopenable and unfixable in the modeler. It is now refused by the builder and by `mxcli check` (MDL-WIDGET27), naming dynamictext. The parity bar started as "match sdk/mpr" and measuring overturned it. ako/TestApp carries three Studio-Pro-authored Forms$StaticImageViewer widgets (FeedbackModule), and legacy disagrees with all three: it omits AlternativeText, which generated/metamodel declares non-optional on both image types, and writes BSON null for the unset Image — measured 0 nulls against 4,400+ empty strings over 40 (type, property) pairs, so an unset by-name reference is "". Legacy's dynamic image was worse: a hand-rolled AlternativeText carrying a FallbackValue key that Forms$ClientTemplate does not have (Fallback / Parameters / Template), four lines after a comment in the shared serializer saying exactly that. sdk/mpr is corrected to match rather than pinned as truth, so both engines assert one shape. Verified: the widget mxcli writes is key-identical to the three Studio Pro references on both engines; the project loads under mxbuild 11.14.0, where the only remaining errors are Mendix's own (CE0582 React deprecation, CE0436 no image selected, CE0489 no image data source — none authorable in MDL), identical on both engines. Each fix has a control: reverting it makes its test fail with the reported symptom. Two divergences are deliberately left: modelsdk writes TabIndex/Width/Height as int32 where Studio Pro writes int64, and omits an empty Widgets list — both project-wide and predating this change, measured on 265 Forms$DivContainer widgets. A third resolves in modelsdk's favour: legacy writes marker 2 for a non-empty Texts$Text list, against 3 in all 3,257 Studio Pro holders. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .claude/skills/fix-issue/findings/sdk.jsonl | 1 + .claude/skills/mendix/custom-widgets/SKILL.md | 5 + cmd/mxcli/syntax/features_page.go | 13 +- docs-site/src/language/widget-types.md | 6 + docs-site/src/reference/capabilities.md | 2 +- .../bug-tests/page-derived-widget-names.mdl | 2 +- .../bug-tests/widgets-deprecated-builtins.mdl | 60 +++++ .../widgets-statictext-unknown-type.fail.mdl | 35 +++ .../modelsdk/widget_child_error_test.go | 10 +- mdl/backend/modelsdk/widget_write.go | 50 +++- .../modelsdk/widget_write_legacy_gaps.go | 224 ++++++++++++++++++ .../modelsdk/widget_write_legacy_gaps_test.go | 206 ++++++++++++++++ .../modelsdk/widget_write_signout_test.go | 11 +- mdl/executor/authoring_language_test.go | 15 +- .../cmd_pages_builder_dataview_test.go | 2 +- mdl/executor/cmd_pages_builder_v3_widgets.go | 46 ++-- mdl/executor/validate_widget_retired.go | 67 ++++++ mdl/executor/validate_widget_retired_test.go | 66 ++++++ mdl/executor/validate_widgets.go | 3 + sdk/mpr/writer_widgets_display.go | 50 +++- sdk/mpr/writer_widgets_image_test.go | 105 ++++++++ 22 files changed, 910 insertions(+), 70 deletions(-) create mode 100644 mdl-examples/bug-tests/widgets-deprecated-builtins.mdl create mode 100644 mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl create mode 100644 mdl/backend/modelsdk/widget_write_legacy_gaps.go create mode 100644 mdl/backend/modelsdk/widget_write_legacy_gaps_test.go create mode 100644 mdl/executor/validate_widget_retired.go create mode 100644 mdl/executor/validate_widget_retired_test.go create mode 100644 sdk/mpr/writer_widgets_image_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index 68d90ce2a..0466f8d82 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -91,3 +91,4 @@ {"area": "mdl/backend", "date": "2026-09-10", "symptom": "SOAP `call web service` writes a document mxbuild accepts and Studio Pro would not have written. A SEND MAPPING is silently DROPPED by both engines \u2014 `send mapping Mod.Export` parses, `mxcli check` passes, `exec` reports success, and nothing in the stored action references the mapping. Operation ARGUMENTS are dropped the same way", "cause": "sdk/mpr.serializeWebServiceCallAction was written without a Studio Pro reference and hardcodes five things it cannot know, and the codec engine's new writer reproduced it deliberately for parity. Measured against three Studio Pro-authored calls in ako/TestApp (Mendix 11.14.0, Clients.GetOrders / GetCustomerOrders / SaveOrder): ServiceName is the WSDL SERVICE name (\"OrdersWS\") not the local part of the imported service's qualified name (\"OrderSoapClient\"); ImportMappingCall.ContentType is \"Xml\" for a SOAP import mapping, not \"Json\"; Range.SingleObject follows cardinality (false for a list) rather than being always true; VariableType is the real result type (DataTypes$ObjectType with an Entity, DataTypes$BooleanType) rather than always DataTypes$VoidType; and a send mapping is Microflows$MappingRequestHandling {ContentType, MappingId, MappingVariableName}. Arguments live in RequestBodyHandling.ParameterMappings as Microflows$WebServiceOperationSimpleParameterMapping entries keyed by an escaped ParameterPath (\"http%3A//www.example.com/:GetOrder|OrderId\")", "file": "`sdk/mpr/writer_microflow_actions.go` (serializeWebServiceCallAction), `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**A guessed type name in a comment becomes a permanent refusal.** Legacy refused send mappings citing `Mendix$AdvancedRequestHandling`, said it 'requires a Studio Pro-generated example to determine the correct type storage name', and that refusal then shipped for as long as nobody went looking. The real type is `Microflows$MappingRequestHandling` \u2014 which THIS CODEBASE ALREADY WRITES for REST result/request handling \u2014 and the guessed name occurs in none of the three reference documents. The lesson is not about SOAP: when a writer refuses because a storage name is unknown, check whether a sibling feature already writes it before treating the refusal as a standing constraint. **Second, and the reason this was found at all: 'no reference exists' is a claim about where you looked.** The parity work asserted that no Studio Pro-authored SOAP document existed to pin against and used that to justify mirroring legacy; one existed in a separate repo the whole time (ako/TestApp, which carries both a consumed client and a published service). Byte-parity with what ships is a legitimate goal for a change scoped to stopping a silent drop \u2014 it is NOT evidence the shape is right, and conflating the two is how six defects got a passing test. Where a reference project exists, name it in the code so the next reader does not repeat the search", "refs": []} {"area": "mdl/backend", "date": "2026-09-11", "symptom": "Making SOAP calls describe structurally instead of as base64 silently rewrote Studio Pro's own documents: a describe -> exec round trip over ako/TestApp flipped `Range.SingleObject` false -> true with no error at all, and turned Clients.SaveOrder's `DataTypes$BooleanType` result into VoidType, which mxbuild reported as `[CE0366]` + `[CE6011]`", "cause": "`webServiceActionRequiresRawBSON` admitted keys BY NAME. That was sound while the supported set was the only nine keys the writer emitted \u2014 but a real call carries FIFTEEN, and six of the new ones are boilerplate mxcli writes at ONE fixed value. Admitting `HttpConfiguration`, `RequestHeaderHandling`, `IsValidationRequired`, `ProxyConfiguration`, `RequestProxyType` and `NewResultHandling` by name meant any call configured beyond mxcli's defaults would be normalised on the next exec", "file": "`mdl/backend/modelsdk/microflow_read_actions.go`, `sdk/mpr/parser_microflow_actions.go` (webServiceActionRequiresRawBSON and its value predicates)", "insight": "**The question a raw-fallback gate answers is not 'do I know this key' but 'would writing this back produce the same document'.** The two coincide only while the writer emits exactly the supported set; the moment a feature lands that widens what is representable, the by-name test starts approving documents it cannot reproduce \u2014 and the loss is invisible, because the result is a VALID model that differs from the user's. **The round trip is the only thing that catches it**: unit tests on the new feature all passed, `mx check` on the newly-written calls was 0 errors, and the regression only appeared when describe -> exec was run over the REFERENCE documents and the BSON diffed (`mxcli bson dump --type microflow --object`, ids normalised away). Two of the five diffs that surfaced were pre-existing microflow describe drift (NoCase case values, bezier control vectors) and unrelated \u2014 worth separating before blaming the change. Fixing it also SHRANK the feature's reach honestly: Studio Pro's own SOAP calls keep the raw form until `Range.SingleObject` is explained, and only mxcli-authored calls describe structurally. **A result type can come from somewhere MDL cannot see** \u2014 SaveOrder's Boolean is the WSDL operation's return type, not an import mapping's entity \u2014 so 'binds a result' does not imply 'derivable'", "refs": []} {"area": "mdl/backend", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec` \u2014 the documented copy operation \u2014 silently turns OFF a microflow's \"apply entity access\". Measured across 342 microflows in 4 projects (11.14.0): every microflow storing `ApplyEntityAccess: true` came back `false` (16/342, all 4 distinct Administration documents). A blocking `show message` also becomes non-blocking (16 microflows). `mxcli check` and mxbuild are both silent \u2014 the model is valid either way", "cause": "TWO causes wearing one symptom. (1) `ApplyEntityAccess` is HARDCODED false in both writers (`mdl/backend/modelsdk/microflow_write.go` SetApplyEntityAccess(false), `sdk/mpr/writer_microflow.go` {Key:\"ApplyEntityAccess\", Value:false}) and `microflows.Microflow` has no field for it, so the read side drops it first. (2) `ShowMessageAction.Blocking` is carried correctly end to end on BOTH engines \u2014 the loss is in DESCRIBE, which has no `blocking` keyword to emit, so the re-parse sets false", "file": "`mdl/backend/modelsdk/microflow_write.go`, `sdk/mpr/writer_microflow.go`, `sdk/microflows/microflows.go` (Microflow struct), `mdl/executor/cmd_microflows_format_action.go` (show message)", "insight": "**A round-trip audit needs a preservation control, or it cannot tell a bug from its own blind spot.** Here it was `StableId`: 342/342 preserved, exactly as ADR-0008 claims \u2014 a method that normalised ids away too aggressively would have reported that as churn, and everything else with it. **Separate `hardcoded in the writer` from `unspellable in DESCRIBE`**: they look identical in a before/after diff and need completely different fixes (model plumbing vs grammar), and `Blocking` proves a property can be perfectly carried by both engines and still be lost by the text round trip. **The same property handled two ways in one codebase is the tell**: `microflows.Rule` carries ApplyEntityAccess correctly (`rule_write.go`, `parser_rule.go`) while `microflows.Microflow` does not \u2014 so this was never an unknown-property gap, just an unfinished one. **Check what an 'empty' value actually holds before calling it a loss**: `ConcurrenyErrorMessage` looked like 58 lost translations and every single one had `Text: \"\"`, i.e. an empty entry versus no entry. Two more traps worth knowing: `ActionActivity.Caption` changes ONLY where `AutoGenerateCaption` is true (a user-set caption survives), and ~90% of the 417-line diff on a real microflow is layout \u2014 bezier vectors, sizes, connection indices \u2014 which buries the two lines that matter", "refs": []} +{"area": "mdl/backend", "date": "2026-09-12", "symptom": "Four plain widget keywords were refused by the DEFAULT engine with \"widget *pages.X not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy\": statictext, dropdown, staticimage, dynamicimage (plus NanoflowSource as a list data source). Nothing in mdl-examples/ or in any skill used those keywords, so the doctype gate ran green the whole time. Then, closing the gap: `statictext` on EITHER engine produced a project that could not be loaded at all — `mx check` aborts with TypeCacheUnknownTypeException for Forms$Text before running any validation, and Studio Pro fails the same way", "cause": "Two unrelated things. (1) The modelsdk widget dispatch simply had no case arms for those five types — reachable set measured at five, not the twenty-three a name scan suggests: nineteen pages.* widget structs are constructed by nothing, and EntityPathSource / ShowHomePageClientAction are written by NEITHER engine, so their fallback named a path legacy could not take either. (2) `Forms$Text` is not a Mendix type. It is absent from modelsdk/gen, from generated/metamodel, and from all 3,257 Studio Pro Texts$Text holders in ako/TestApp; `buildTextWidgetV3` minted it for both `text` and `statictext`. Writing it is worse than a build error — the project is unopenable and unfixable in the modeler", "file": "`mdl/backend/modelsdk/widget_write_legacy_gaps.go` (new), `mdl/backend/modelsdk/widget_write.go`, `sdk/mpr/writer_widgets_display.go`, `mdl/executor/validate_widget_retired.go` (new), `mdl/executor/cmd_pages_builder_v3_widgets.go`", "insight": "**\"Parity with the other engine\" is the wrong bar until you have checked whether the other engine is right.** That was the starting assumption here and measuring overturned it: ako/TestApp carries three Studio-Pro-authored Forms$StaticImageViewer widgets (FeedbackModule) and legacy disagreed with all three — it omitted AlternativeText (non-omitempty in generated/metamodel) and wrote BSON null for the unset Image. Legacy's dynamic image was worse: a hand-rolled AlternativeText carrying a FallbackValue key that Forms$ClientTemplate does not have, four lines after a comment in the SHARED serializer saying exactly that (\"Must be Fallback object, not FallbackValue string\"). sdk/mpr was corrected to match rather than pinned as truth. **Look for a Studio Pro reference inside the fixture before concluding there is none** — `bson.Unmarshal` every mprcontents unit and count instances of the $Type; the marketplace modules a blank app ships are Studio Pro output. The same 40-line scanner settles value-shape questions no doc answers: measured 0 nulls against 4,400+ empty strings for by-name references, so an unset QualifiedName is \"\", never null; and marker 3 in 3,257 of 3,257 Texts$Text lists, so legacy's marker 2 for a non-empty one is the wrong side of that divergence, not modelsdk's 3. **A widget keyword the example corpus never uses is a keyword nothing tests**, whatever the coverage numbers say. **Check that the project still LOADS, not just that it builds** — a load failure aborts `mx check` before the error list, so grepping for CE-codes finds nothing and reads like success", "refs": []} diff --git a/.claude/skills/fix-issue/findings/sdk.jsonl b/.claude/skills/fix-issue/findings/sdk.jsonl index 2e6fd9f91..8a98c1a2d 100644 --- a/.claude/skills/fix-issue/findings/sdk.jsonl +++ b/.claude/skills/fix-issue/findings/sdk.jsonl @@ -40,3 +40,4 @@ {"area": "sdk/mpr", "date": "2026-08-31", "symptom": "A document mxcli writes carries a different **typed-array marker** (the leading `int32` of a Mendix array) than the equivalent Studio Pro document — e.g. every list in a `CREATE OR REPLACE NAVIGATION` profile was `1` where Studio Pro writes `2` or `3`. No error, no warning, no build failure: it renders and opens", "cause": "The writers hand-build `bson.A{int32(1)}` per list. The marker is a **per-field constant**, not a function of the list's contents (`Forms$FormSettings.ParameterMappings` is `2` in 816 empty and 306 non-empty documents alike), so it cannot be derived — it has to be read off real documents", "file": "`sdk/mpr/writer_navigation.go` + `mdl/backend/modelsdk/navigation_write.go` + `modelsdk/mpr/nav_patch.go` (`navMarker*` / `navpMarker*` constants), `mdl/backend/modelsdk/navigation_profile_add.go`, `modelsdk/codec/defaults.go` (`RegisterListMarker`) for the codec paths", "insight": "**Census, don't reason.** Walk every `.mxunit` on the machine, tabulate `(parent $Type, field, marker, empty?)`, and take the value the Studio Pro documents carry — 19,078 files across 54 projects settled five of six navigation fields outright. **`int32(1)` is NOT invalid**, whatever `debug-bson.md` used to say: a Marketplace `.mpk` mxcli has never touched uses it for `CustomWidgets$WidgetValueType.AllowedTypes` (212k occurrences) and `Forms$Page.AllowedModuleRoles`. Believing otherwise turns a per-field mismatch into a phantom corruption bug and sends the fix in the wrong direction. Where the census has no observation, **find a document that has one** rather than picking: `HomeItems` was `2` in all 51 stored profiles but every one was empty, and `navigation_profile_add.go` wrote `3` from a PED session that could not be re-run. ako/TestApp settled it — a Studio Pro-authored profile whose `HomeItems` holds two `Navigation$RoleBasedHomePage` elements at marker **2**, the non-empty case the census could not reach. One project with the feature actually configured beats any amount of reasoning about empty lists. Verify by dumping the written document and the project's own pristine reference and diffing the marker column, not by `mx check`, which is silent on all of it"} {"area": "sdk", "date": "2026-09-04", "symptom": "REPORTED AS A BUG, MEASURED AS A NON-BUG. `create association \u2026 type ReferenceSet owner Both` without `STORAGE TABLE` writes `StorageFormat: \"Column\"`, which was reported as \"not a legal many-to-many\" and worked around by respelling every such association.", "cause": "Nothing is broken. Measured on Mendix 11.13 against a live PostgreSQL, with the two spellings side by side in one app: `App.PA_PB` (ReferenceSet, StorageFormat Column) and `App.PC_PD` (ReferenceSet, StorageFormat Table) produce IDENTICAL DDL \u2014 `app$pa_pb(app$paid, app$pbid)` and `app$pc_pd(app$pcid, app$pdid)`, two FK constraints each. The app boots and serves HTTP 200, and `mx check` reports 0 errors. Mendix ignores StorageFormat for a reference set and always uses a junction table.", "file": "no code change \u2014 `mdl/executor/cmd_associations.go` defaults storageFormat to Column for every association type, and that is harmless", "insight": "A reported bug is a symptom plus an EXPLANATION, and the explanation is the part to re-measure. \"mxcli writes Column\" was true; \"which is not a legal many-to-many\" was the inference, and it cost the reporter a rewrite of ten associations. The cheap discriminator was to author BOTH spellings in one app and compare the DDL the runtime actually creates \u2014 a side-by-side control in the same boot, rather than reasoning about what a column could hold. Note the pkill trap from the same FINDINGS (\u00a711) applies when tidying up afterwards: `pkill -f \"mxcli run\"` matches the calling shell and kills it (exit 144)."} {"area": "sdk/mpr", "date": "2026-09-06", "symptom": "`mx check` reported CE0066 \"Entity access is out of date\" at \"Domain model of module 'BusinessEvents'\" after `create or modify persistent entity BusinessEvents.PublishedBusinessEvent ( EventId: long )` over the real BusinessEvents 3.12.0 marketplace module. LEGACY ENGINE ONLY — the codec engine produced 0 errors from the same script. Caught by the integration gate (TestMxCheck_DoctypeScripts/13-business-events-examples.mdl/legacy), not by any unit test.", "cause": "ReconcileMemberAccesses in sdk/mpr/writer_security.go skipped any rule whose MemberAccesses list held only the storage marker (`if len(maArr) <= 1 { break }`), so it never topped one up. A rule with zero member entries on an entity that HAS members is precisely the out-of-date state CE0066 names, so the skip left behind the one thing the function exists to prevent. Nothing reached that state until `create or modify entity` started PRESERVING access rules instead of deleting them: the rewrite dropped all five attributes the Administrator rule covered, the prune emptied the list, and the new EventId then never got an entry. Fixed by narrowing the guard to `len(maArr) == 0` (no storage marker at all).", "file": "`sdk/mpr/writer_security.go` (ReconcileMemberAccesses, the MemberAccesses loop); tests `sdk/mpr/writer_security_reconcile_test.go`", "insight": "A fix that starts PRESERVING something reaches states no prior code could produce, so its blast radius is every consumer of that thing — here a reconcile function untouched for months. The engine split is the tell worth acting on: identical script, 0 errors on modelsdk and CE0066 on legacy, which localises the defect to the legacy path in one measurement and makes the codec engine the reference for what the document should contain (dumped both: 1 member entry vs 0). Also note where this was caught — only the integration gate exercises a real marketplace module, and only that module had a rule whose entire member set the script drops. The unit tests written for the entity fix were green throughout, and were right to be: the entity layer did exactly what it should. Keep the empty-list case as a named test on both sides, with a member-less entity as the control, since the old guard covered that case by accident and removing it must not turn every member-less entity into a write."} +{"area": "sdk/mpr", "date": "2026-09-12", "symptom": "The legacy writer's image widgets disagree with Studio Pro. `serializeStaticImage` omits AlternativeText entirely; `serializeDynamicImage` writes one containing a `FallbackValue` string; both write BSON null for the unset Image / DefaultImage. mxbuild accepts all of it at 0 errors", "cause": "`Forms$ClientTemplate` has exactly three properties — Fallback (Texts$Text), Parameters, Template (generated/metamodel, and all three Studio Pro references). The dynamic image hand-rolled its own holder instead of calling `serializeClientTemplate`, and invented FallbackValue. AlternativeText is declared without omitempty on both image types and appears in 3/3 references, so omitting it is a drop, not an optional key", "file": "`sdk/mpr/writer_widgets_display.go` (serializeStaticImage, serializeDynamicImage, emptyAlternativeText)", "insight": "**A hand-rolled copy of a shared serializer is where the invented key lives.** The correct helper was four lines away and carried a comment naming this exact mistake; the copy still got it wrong, because nothing compares the two. Grep for a type's $Type string and check whether every construction site goes through one builder. **mxbuild is not a check for this class at all** — it tolerates unknown properties, while Studio Pro resolves every stored property against the type's property list and throws \"Sequence contains no matching element\" at MprProperty.cs. The available substitutes are generated/metamodel (the arbiter) and a real Studio Pro document from a marketplace module in the fixture", "refs": []} diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index 95accdd08..f45ff400f 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -80,6 +80,11 @@ A name resolving to no installed definition is an **error** (MDL-WIDGET25, with near-miss suggestions), and a container the parent does not declare is MDL-WIDGET26. Both need `-p`: without a project, mxcli knows only its embedded widgets, so it stays quiet rather than reporting every real widget as unknown. + +MDL-WIDGET27 needs no project: `statictext` writes `Forms$Text`, a type Mendix +does not have, and the project that comes out cannot be *loaded* at all (`mx +check` and Studio Pro both stop at `TypeCacheUnknownTypeException` before +validation). Use `dynamictext` with a literal `Content:`. If a widget you have installed is not found, extract its definition: ```bash diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 3b59eebac..6ba4de889 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -137,15 +137,16 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "-- `check --references` rather than failing the build with CE1613.\n" + "-- The alternatives are the URL form above, or `ImageType: icon`.\n\n" + "-- Any pluggable widget by its id (id FIRST, then the name)\nPLUGGABLEWIDGET 'com.mendix.widget.web.badge.Badge' name (value: 'x')\nCUSTOMWIDGET 'com.mendix.widget.custom.x.X' name (prop: 'x') -- legacy spelling\n\n" + - "-- Accepted by the parser, NOT writable on the default engine.\n" + - "-- Measured on 11.13.0: each is refused with\n" + - "-- \"widget *pages.X not yet supported by the modelsdk engine\"\n" + - "-- Re-run with MXCLI_ENGINE=legacy, or use the alternative given:\n" + - "-- STATICTEXT -> DYNAMICTEXT with a literal Content\n" + + "-- Deprecated in the Mendix 11 React client. These are written correctly by\n" + + "-- both engines, but mxbuild reports CE0582 (\"not supported in React client\")\n" + + "-- on each, so prefer the alternative:\n" + "-- STATICIMAGE -> IMAGE\n" + "-- DYNAMICIMAGE -> IMAGE\n" + "-- DROPDOWN -> COMBOBOX\n" + - "-- And two the executor refuses on BOTH engines, each with its own message:\n" + + "-- And three the executor refuses on BOTH engines, each with its own message:\n" + + "-- STATICTEXT (writes Forms$Text, a type Mendix no longer has — the\n" + + "-- project could not be OPENED afterwards; MDL-WIDGET27.\n" + + "-- Use DYNAMICTEXT with a literal Content.)\n" + "-- REFERENCESELECTOR (unsupported widget type)\n" + "-- LEGACYDATAGRID (use DATAGRID for the pluggable equivalent on Mendix 11+)", Example: "DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n COMBOBOX cbStatus (Label: 'Status', Attribute: Status)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n}", diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index ed29b9325..ce184227f 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -508,6 +508,12 @@ accepted widget — MDL-WIDGET25, with the nearest known names suggested. A container keyword the parent widget does not declare is MDL-WIDGET26. Both need a project open (`-p`), since without one mxcli knows only its embedded widgets. +A third, MDL-WIDGET27, needs no project: `statictext` writes `Forms$Text`, and +Mendix has no such type. That is not a build error but a **load** error — `mx +check` and Studio Pro both stop at `TypeCacheUnknownTypeException` before any +validation runs, so the page cannot even be opened to repair it. Use +`DYNAMICTEXT` with a literal `Content:`. + If a widget you have installed is not found, its definition has not been extracted yet: diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 4463722b8..6b46b2df6 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -64,7 +64,7 @@ Everything mxcli can do, organized by use case. | Modify page | `ALTER PAGE ... SET/INSERT/DROP/REPLACE` | In-place modifications | | Built-in widgets | TEXTBOX, TEXTAREA, DATEPICKER, etc. | All standard widgets | | Layout widgets | LAYOUTGRID, CONTAINER, GROUPBOX | With responsive columns | -| Display widgets | DYNAMICTEXT, STATICTEXT, IMAGE | Including pluggable Image | +| Display widgets | DYNAMICTEXT, IMAGE | Including pluggable Image | | Data widgets | DATAVIEW, LISTVIEW | With datasource binding | | Pluggable widgets | DATAGRID2, GALLERY, COMBOBOX | Template-based | | Action buttons | ACTIONBUTTON | Save, cancel, microflow, page | diff --git a/mdl-examples/bug-tests/page-derived-widget-names.mdl b/mdl-examples/bug-tests/page-derived-widget-names.mdl index f989419a8..d9141bd80 100644 --- a/mdl-examples/bug-tests/page-derived-widget-names.mdl +++ b/mdl-examples/bug-tests/page-derived-widget-names.mdl @@ -46,7 +46,7 @@ create or modify page DerivedNames.Overview LAYOUTGRID lgFirst { ROW row1 { COLUMN col1 (DesktopWidth: AutoFill) { - STATICTEXT stFirst (Content: 'First grid') + DYNAMICTEXT stFirst (Content: 'First grid') } } } diff --git a/mdl-examples/bug-tests/widgets-deprecated-builtins.mdl b/mdl-examples/bug-tests/widgets-deprecated-builtins.mdl new file mode 100644 index 000000000..5c4c96fdd --- /dev/null +++ b/mdl-examples/bug-tests/widgets-deprecated-builtins.mdl @@ -0,0 +1,60 @@ +-- The three built-in widgets that used to send a user to MXCLI_ENGINE=legacy. +-- +-- `dropdown`, `staticimage` and `dynamicimage` were refused by the default +-- (modelsdk) engine with "widget *pages.X not yet supported by the modelsdk +-- engine", which is one of the reasons the legacy engine could not be retired. +-- Nothing in mdl-examples/ or in any skill used the keywords, so the doctype +-- gate ran green while four plain widget keywords failed on the default engine. +-- +-- All three are DEPRECATED in the Mendix 11 React client — mxbuild reports +-- CE0582 on each and Studio Pro offers to convert them — so this file is not in +-- doctype-tests/, which is the "should be used" corpus. Prefer COMBOBOX and the +-- pluggable IMAGE. mxcli still has to write them correctly, because a project +-- being converted UP already contains them. +-- +-- The BSON is pinned by unit tests, not by this file: `check` never serializes. +-- See mdl/backend/modelsdk/widget_write_legacy_gaps_test.go and +-- sdk/mpr/writer_widgets_image_test.go, which assert the same shape for both +-- engines against the three Studio-Pro-authored Forms$StaticImageViewer widgets +-- in ako/TestApp. + +create or modify module WidgetProbe; + +create or modify enumeration WidgetProbe.Status ( + Open 'Open', + Closed 'Closed' +); + +create or modify entity WidgetProbe.Ticket ( + Title: string(200), + Status: enum WidgetProbe.Status +); + +create or replace page WidgetProbe.PDeprecatedWidgets +( + params: { $Ticket: WidgetProbe.Ticket }, + title: 'Deprecated built-in widgets', + layout: Atlas_Core.Atlas_Default +) +{ + layoutgrid lg1 { row row1 { column col1 (desktopwidth: autofill) { + dataview dvTicket (datasource: $Ticket) { + textbox txtTitle (label: 'Title', attribute: Title) + + -- Forms$DropDown. The pluggable COMBOBOX is the Mendix 11 widget. + dropdown ddStatus (label: 'Status', attribute: Status) + + -- Forms$StaticImageViewer. MDL cannot name the image (there is no `Image:` + -- on this widget), so it always writes the unset value — which is the empty + -- string, not null. mxbuild reports CE0436 "No image selected." for that, + -- inherently and on both engines. + staticimage imgStatic (width: 120, height: 80) + + -- Forms$ImageViewer, whose DataSource is a Forms$ImageViewerSource. MDL + -- cannot bind that source's entity path either, so mxbuild reports CE0489 + -- "Select an entity for the data source of this dynamic image" — again on + -- both engines, and again a missing property rather than a wrong one. + dynamicimage imgDynamic (width: 120, height: 80) + } + }}} +} diff --git a/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl b/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl new file mode 100644 index 000000000..abc8f7525 --- /dev/null +++ b/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl @@ -0,0 +1,35 @@ +-- `statictext` writes a type Mendix does not have (refusal, MDL-WIDGET27). +-- +-- `mxcli check mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl` +-- reports MDL-WIDGET27. Before the fix this file checked clean, executed, and +-- produced a project that could not be LOADED at all: +-- +-- ERROR: System.AggregateException: One or more errors occurred. +-- (The type cache does not contain a type with qualified name Forms$Text.) +-- ---> Mendix.Modeler.Storage.Caches.TypeCacheUnknownTypeException +-- +-- Not a build error — a load error. `mx check` stops there before running any +-- validation, and Studio Pro fails the same way, so the page cannot be repaired +-- in the modeler. Measured on mxbuild 11.14.0 against ako/TestApp, and +-- reproduced identically under MXCLI_ENGINE=legacy: both writers emitted the +-- type, so switching engines was never a workaround. +-- +-- Forms$Text is absent from modelsdk/gen, from generated/metamodel, and from +-- all 3,257 Studio Pro text holders in ako/TestApp. Forms$DynamicText is the +-- widget Mendix has. + +create or modify module WidgetProbe; + +-- REFUSED. One keyword, one unopenable project. +create or replace page WidgetProbe.PStaticText ( title: 'S', layout: Atlas_Core.Atlas_Default ) +{ + statictext t1 (content: 'hello') +} +/ + +-- CONTROL: the replacement the refusal names. Without this the file would pass +-- just as well against a rule that refused every text widget. +create or replace page WidgetProbe.PDynamicText ( title: 'D', layout: Atlas_Core.Atlas_Default ) +{ + dynamictext t2 (content: 'hello') +} diff --git a/mdl/backend/modelsdk/widget_child_error_test.go b/mdl/backend/modelsdk/widget_child_error_test.go index 3f01c9252..0deba44de 100644 --- a/mdl/backend/modelsdk/widget_child_error_test.go +++ b/mdl/backend/modelsdk/widget_child_error_test.go @@ -21,9 +21,9 @@ func TestChildSerializeErr_RecordedAndDrained(t *testing.T) { t.Fatalf("accumulator not empty at start: %v", err) } - // A nanoflow datasource is not yet representable by the codec engine. + // A listen-to-widget datasource is not yet representable by the codec engine. got := codecChildSerializer{}.SerializeCustomWidgetDataSource( - &pages.NanoflowSource{Nanoflow: "M.GetOrders"}) + &pages.ListenToWidgetSource{}) if got != nil { t.Errorf("unsupported datasource serialized to %v, want nil", got) } @@ -32,7 +32,7 @@ func TestChildSerializeErr_RecordedAndDrained(t *testing.T) { if err == nil { t.Fatal("failure was not recorded; it would be dropped silently") } - if !strings.Contains(err.Error(), "NanoflowSource") { + if !strings.Contains(err.Error(), "ListenToWidgetSource") { t.Errorf("error does not name the construct: %v", err) } @@ -69,7 +69,7 @@ func TestCreatePage_FailsOnDroppedChild(t *testing.T) { // Simulate the executor building a widget tree whose child could not be // serialized, exactly as SerializeCustomWidgetDataSource does above. - codecChildSerializer{}.SerializeCustomWidgetDataSource(&pages.NanoflowSource{Nanoflow: "M.GetOrders"}) + codecChildSerializer{}.SerializeCustomWidgetDataSource(&pages.ListenToWidgetSource{}) page := &pages.Page{Name: "DropProbe"} page.ID = model.ID("") @@ -77,7 +77,7 @@ func TestCreatePage_FailsOnDroppedChild(t *testing.T) { if err == nil { t.Fatal("CreatePage succeeded after a child was dropped") } - if !strings.Contains(err.Error(), "NanoflowSource") { + if !strings.Contains(err.Error(), "ListenToWidgetSource") { t.Errorf("error does not explain what was dropped: %v", err) } } diff --git a/mdl/backend/modelsdk/widget_write.go b/mdl/backend/modelsdk/widget_write.go index 773e157a6..49e4f99dd 100644 --- a/mdl/backend/modelsdk/widget_write.go +++ b/mdl/backend/modelsdk/widget_write.go @@ -710,8 +710,27 @@ func widgetToGen(w pages.Widget) (element.Element, error) { case *pages.CustomWidget: return customWidgetToGen(x) + // The widgets that used to send a user to the legacy engine, none of them + // covered by the doctype gate — see widget_write_legacy_gaps.go. + // + // pages.Text (Forms$Text) is deliberately NOT here. Mendix has no such type, + // so writing one makes the project unopenable; the keyword that built it is + // refused (mdl/executor/validate_widget_retired.go) and nothing constructs + // the struct any more. An old project that carries one keeps it because + // ALTER PAGE mutates the stored gen document rather than rebuilding from the + // semantic model — this switch is never asked about it. + case *pages.DropDown: + return dropDownToGen(x) + + case *pages.StaticImage: + return staticImageToGen(x) + + case *pages.DynamicImage: + return dynamicImageToGen(x) + default: - return nil, fmt.Errorf("CreatePage: widget %T not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy", w) + return nil, fmt.Errorf("CreatePage: widget %T is not supported by either engine — "+ + "please file an issue with the MDL that produced it", w) } } @@ -1214,6 +1233,13 @@ func dataViewSourceToGen(ds pages.DataSource) (element.Element, error) { ms.SetMicroflowSettings(microflowSettingsToGen(d.Microflow, d.ParameterMappings)) return ms, nil + // A NANOFLOW data source. Its sibling above goes through gen; this one is + // built raw because gen binds the nanoflow name directly on the source while + // Studio Pro nests it in a Forms$NanoflowSettings child — see + // nanoflowSourceToGen. + case *pages.NanoflowSource: + return nanoflowSourceToGen(d), nil + case *pages.AssociationSource: // A DataView showing a to-one referenced object ("data from context over // an association") is a Forms$DataViewSource whose EntityRef is an @@ -1223,7 +1249,7 @@ func dataViewSourceToGen(ds pages.DataSource) (element.Element, error) { return dataViewContextAssociationSourceToGen(d), nil default: - return nil, fmt.Errorf("CreatePage: DataView source %T not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy", ds) + return nil, fmt.Errorf("CreatePage: DataView source %T is not supported by either engine — please file an issue", ds) } } @@ -1306,10 +1332,17 @@ func listViewSourceToGen(ds pages.DataSource) (element.Element, error) { ms.SetForceFullObjects(false) ms.SetMicroflowSettings(microflowSettingsToGen(d.Microflow, d.ParameterMappings)) return ms, nil + + // A NANOFLOW data source. Its sibling above goes through gen; this one is + // built raw because gen binds the nanoflow name directly on the source while + // Studio Pro nests it in a Forms$NanoflowSettings child — see + // nanoflowSourceToGen. + case *pages.NanoflowSource: + return nanoflowSourceToGen(d), nil case *pages.AssociationSource: return associationSourceToGen(d), nil default: - return nil, fmt.Errorf("CreatePage: ListView source %T not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy", ds) + return nil, fmt.Errorf("CreatePage: ListView source %T is not supported by either engine — please file an issue", ds) } } @@ -1361,11 +1394,18 @@ func customWidgetDataSourceToGen(ds pages.DataSource) (element.Element, error) { ms.SetMicroflowSettings(microflowSettingsToGen(d.Microflow, d.ParameterMappings)) return ms, nil + // A NANOFLOW data source. Its sibling above goes through gen; this one is + // built raw because gen binds the nanoflow name directly on the source while + // Studio Pro nests it in a Forms$NanoflowSettings child — see + // nanoflowSourceToGen. + case *pages.NanoflowSource: + return nanoflowSourceToGen(d), nil + case *pages.AssociationSource: return associationSourceToGen(d), nil default: - return nil, fmt.Errorf("modelsdk: pluggable widget data source %T not yet supported — rerun with MXCLI_ENGINE=legacy", ds) + return nil, fmt.Errorf("modelsdk: pluggable widget data source %T is not supported by either engine — please file an issue", ds) } } @@ -1609,7 +1649,7 @@ func clientActionToGen(a pages.ClientAction) (element.Element, error) { g.SetPageSettings(formSettingsToGen(x.PageName)) return g, nil default: - return nil, fmt.Errorf("CreatePage: client action %T not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy", a) + return nil, fmt.Errorf("CreatePage: client action %T is not supported by either engine — please file an issue", a) } } diff --git a/mdl/backend/modelsdk/widget_write_legacy_gaps.go b/mdl/backend/modelsdk/widget_write_legacy_gaps.go new file mode 100644 index 000000000..314f7387c --- /dev/null +++ b/mdl/backend/modelsdk/widget_write_legacy_gaps.go @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + genPg "github.com/mendixlabs/mxcli/modelsdk/gen/pages" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// The last widgets that sent a user to `MXCLI_ENGINE=legacy`. +// +// Every "not yet supported by the modelsdk engine" message is a reason the +// legacy engine has to stay shipped and tested. Measured against sdk/pages, the +// reachable set was five, not the twenty-three a name-based scan suggests: +// nineteen widget structs are never constructed by the executor at all, and two +// of the three remaining data-source / client-action gaps (EntityPathSource, +// ShowHomePageClientAction) are written by NEITHER engine and built by nothing, +// so the fallback message named a path legacy could not take either. +// +// The five that were real, each confirmed by running `mxcli exec` on a page that +// uses them: +// +// statictext -> pages.Text Forms$Text +// dropdown -> pages.DropDown Forms$DropDown +// staticimage -> pages.StaticImage Forms$StaticImageViewer +// dynamicimage -> pages.DynamicImage Forms$ImageViewer +// -> pages.NanoflowSource Forms$NanoflowSource (list/grid data source) +// +// None was covered by the doctype gate: no example in mdl-examples/ or in any +// skill uses those keywords, which is why the suite ran green on modelsdk while +// four plain widget keywords failed. +// +// The first of the five turned out not to be a gap at all. Closing it and +// running the result through `mx check` showed that BOTH engines wrote a project +// that could not be LOADED — Mendix has no Forms$Text — so `statictext` is now +// refused at build and check time (MDL-WIDGET27, +// mdl/executor/validate_widget_retired.go). Nothing constructs pages.Text any +// more, so there is no writer for it here either — an old project that carries +// one keeps it because ALTER PAGE mutates the stored gen document rather than +// rebuilding from the semantic model. +// +// The bar here is the Studio Pro reference, NOT parity with sdk/mpr. That was +// the starting assumption and measuring it overturned it: ako/TestApp carries +// three Studio-Pro-authored Forms$StaticImageViewer widgets (FeedbackModule), +// and legacy disagrees with all three in two ways — it omits AlternativeText +// (which generated/metamodel declares non-optional on both image types) and it +// writes BSON null for the unset Image. Across the whole corpus a by-name +// reference is written as an EMPTY STRING and never as null: 0 nulls against +// 4,400+ empty strings over 40 (type, property) pairs. Legacy's dynamic image is +// worse still — its hand-rolled AlternativeText carries a "FallbackValue" string +// that Forms$ClientTemplate does not have (metamodel: Fallback / Parameters / +// Template), which is the invent-a-key defect from CLAUDE.md, invisible to +// mxbuild. sdk/mpr was corrected to match rather than pinned as ground truth, so +// the two test files can assert one shape for both engines +// (widget_write_legacy_gaps_test.go and sdk/mpr/writer_widgets_image_test.go). +// +// Two divergences are deliberately left alone, because measuring them showed +// they are project-wide and predate this file — fixing them here would hide +// them: modelsdk writes TabIndex/Width/Height as int32 where Studio Pro and +// legacy write int64 (measured on 265 Forms$DivContainer widgets, not just +// these), and modelsdk omits an empty Widgets list where legacy writes [3]. + +// dropDownToGen builds a Forms$DropDown (an enumeration/association selector). +func dropDownToGen(dd *pages.DropDown) (element.Element, error) { + g := genPg.NewDropDown() + applyWidgetBase(g, &dd.BaseWidget) + g.SetAriaRequired(false) + if ref := attributeRefToGen(dd.AttributePath); ref != nil { + g.SetAttributeRef(ref) + } + g.SetEditable(pages.WidgetEditability(&dd.BaseWidget)) + // An empty Texts$Text, not a null: the property is the caption of the blank + // option and Studio Pro always writes the holder. + g.SetEmptyOptionCaption(emptyTranslatedText()) + if dd.Label != "" { + g.SetLabelTemplate(textAsClientTemplate(textFromString(dd.Label))) + } + onChange, err := clientActionToGen(dd.OnChangeAction) + if err != nil { + return nil, err + } + g.SetOnChangeAction(onChange) + g.SetOnEnterAction(noActionGen()) + g.SetOnLeaveAction(noActionGen()) + g.SetReadOnlyStyle("Inherit") + g.SetValidation(widgetValidationToGen()) + return g, nil +} + +// staticImageToGen builds a Forms$StaticImageViewer. +// +// Deprecated in the Mendix 11 React client (CE0582) — `image` routes to the +// pluggable widget instead — but `staticimage` is still a keyword the executor +// dispatches, so the writer has to answer for it. Unlike `statictext` the TYPE +// exists: the project loads, and CE0582 is Mendix's own advice rather than a +// defect, so refusing it would be over-reach. +func staticImageToGen(img *pages.StaticImage) (element.Element, error) { + g := genPg.NewStaticImageViewer() + applyWidgetBase(g, &img.BaseWidget) + g.SetAlternativeText(emptyClientTemplate()) + click, err := clientActionToGen(img.OnClickAction) + if err != nil { + return nil, err + } + g.SetClickAction(click) + // MDL cannot name an image (the builder never fills ImageID), so this is + // always the unset value — and unset is "", not null; see the header. + g.SetImageQualifiedName("") + g.SetHeight(int32(img.Height)) + g.SetHeightUnit("Auto") + g.SetResponsive(img.Responsive) + g.SetWidth(int32(img.Width)) + g.SetWidthUnit("Auto") + return g, nil +} + +// dynamicImageToGen builds a Forms$ImageViewer — gen calls the type +// DynamicImageViewer, and its storage name is the one that matters. +func dynamicImageToGen(img *pages.DynamicImage) (element.Element, error) { + g := genPg.NewDynamicImageViewer() + applyWidgetBase(g, &img.BaseWidget) + g.SetAlternativeText(emptyClientTemplate()) + click, err := clientActionToGen(img.OnClickAction) + if err != nil { + return nil, err + } + g.SetClickAction(click) + g.SetDataSource(imageViewerSourceToGen()) + g.SetDefaultImageQualifiedName("") + g.SetHeight(int32(img.Height)) + g.SetHeightUnit("Auto") + g.SetOnClickEnlarge(false) + g.SetResponsive(img.Responsive) + g.SetShowAsThumbnail(false) + g.SetWidth(int32(img.Width)) + g.SetWidthUnit("Auto") + return g, nil +} + +// imageViewerSourceToGen builds the empty Forms$ImageViewerSource a dynamic +// image carries when no entity path has been set. +func imageViewerSourceToGen() element.Element { + src := genPg.NewImageViewerSource() + assignID(src) + return src +} + +// nanoflowSourceToGen builds a Forms$NanoflowSource — a list widget's "nanoflow" +// data source. +// +// Built raw, and this one is a judgement rather than a limitation: gen's +// NanoflowSource offers ForceFullObjects and NanoflowQualifiedName, binding the +// nanoflow name DIRECTLY on the source, while Studio Pro nests it inside a +// Forms$NanoflowSettings child alongside ParameterMappings — which is what +// sdk/mpr writes. Writing gen's shape would put the name in a key Studio Pro +// does not read there, the same class of defect as the storage-name overrides +// (CLAUDE.md). Legacy's shape is the one with a working project behind it. +func nanoflowSourceToGen(d *pages.NanoflowSource) element.Element { + g := newElem("Forms$NanoflowSource", string(d.ID)) + settings := newElem("Forms$NanoflowSettings", "") + addStr(settings, "Nanoflow", d.Nanoflow) + addEmptyTypedList(settings, "ParameterMappings", 3) + addPart(g, "NanoflowSettings", settings) + return g +} + +// emptyClientTemplate is the Forms$ClientTemplate an image's AlternativeText +// carries when no alt text has been set: an empty Template, an empty Fallback +// and no parameters. Matches the three Studio-Pro-authored StaticImageViewer +// widgets in ako/TestApp element for element. +func emptyClientTemplate() element.Element { + return textAsClientTemplate(nil) +} + +// emptyTranslatedText is a Texts$Text with no translations — the holder Studio +// Pro writes for an unset caption. +func emptyTranslatedText() element.Element { + holder := newElem("Texts$Text", "") + addEmptyTypedList(holder, "Items", 3) + return holder +} + +func init() { + // Every widget here needs its null slots and its list marker registered, or + // it serializes with keys missing and under the wrong array version. Both + // were caught by diffing the two engines' output for one page carrying all + // four widgets — neither shows up as a build error. + // + // Marker 2, not the codec's default of 3: a container's Widgets list takes + // its marker from the CHILD type, so an unregistered widget silently changed + // the marker of the list it sits in. + codec.RegisterTypeDefaults("Forms$DropDown", codec.TypeDefaults{ + NullFields: []string{ + "AttributeRef", "ScreenReaderLabel", "SourceVariable", "LabelTemplate", + "ConditionalVisibilitySettings", "ConditionalEditabilitySettings", + "NativeAccessibilitySettings", + }, + }) + codec.RegisterListMarker("Forms$DropDown", 2) + + // Image / DefaultImage are by-name references and are set to "" above, not + // listed here: an unset one is an empty string in every Studio Pro document + // measured. Only the two child slots are genuinely null. + codec.RegisterTypeDefaults("Forms$StaticImageViewer", codec.TypeDefaults{ + NullFields: []string{ + "ConditionalVisibilitySettings", "NativeAccessibilitySettings", + }, + }) + codec.RegisterListMarker("Forms$StaticImageViewer", 2) + + codec.RegisterTypeDefaults("Forms$ImageViewer", codec.TypeDefaults{ + NullFields: []string{ + "ConditionalVisibilitySettings", "NativeAccessibilitySettings", + }, + }) + codec.RegisterListMarker("Forms$ImageViewer", 2) + + // EntityRef is the attribute path the image comes from — null when unbound. + codec.RegisterTypeDefaults("Forms$ImageViewerSource", codec.TypeDefaults{ + NullFields: []string{"EntityRef"}, + }) +} diff --git a/mdl/backend/modelsdk/widget_write_legacy_gaps_test.go b/mdl/backend/modelsdk/widget_write_legacy_gaps_test.go new file mode 100644 index 000000000..6cb46a01b --- /dev/null +++ b/mdl/backend/modelsdk/widget_write_legacy_gaps_test.go @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "sort" + "strings" + "testing" + + bsonv1 "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// The widgets that used to send a user to MXCLI_ENGINE=legacy, pinned to the +// shape Mendix actually stores. +// +// studioProStaticImageKeys is measured, not derived: the three +// Forms$StaticImageViewer widgets ako/TestApp inherits from FeedbackModule are +// Studio Pro's own, and all three carry exactly these keys. The others come from +// generated/metamodel, which CLAUDE.md names the arbiter — for each type it is +// every property the metamodel declares without `omitempty`, plus the optional +// ones mxcli sets. +// +// The same expectations are asserted against the legacy writer in +// sdk/mpr/writer_widgets_display_test.go, so the two engines cannot drift apart +// silently. They already had: legacy omitted AlternativeText here and wrote a +// FallbackValue key that Forms$ClientTemplate does not have. +var ( + studioProStaticImageKeys = []string{ + "$ID", "$Type", "AlternativeText", "Appearance", "ClickAction", + "ConditionalVisibilitySettings", "Height", "HeightUnit", "Image", "Name", + "NativeAccessibilitySettings", "Responsive", "TabIndex", "Width", "WidthUnit", + } + dynamicImageKeys = []string{ + "$ID", "$Type", "AlternativeText", "Appearance", "ClickAction", + "ConditionalVisibilitySettings", "DataSource", "DefaultImage", "Height", + "HeightUnit", "Name", "NativeAccessibilitySettings", "OnClickEnlarge", + "Responsive", "ShowAsThumbnail", "TabIndex", "Width", "WidthUnit", + } + dropDownKeys = []string{ + "$ID", "$Type", "Appearance", "AriaRequired", "AttributeRef", + "ConditionalEditabilitySettings", "ConditionalVisibilitySettings", "Editable", + "EmptyOptionCaption", "LabelTemplate", "Name", "NativeAccessibilitySettings", + "OnChangeAction", "OnEnterAction", "OnLeaveAction", "ReadOnlyStyle", + "ScreenReaderLabel", "SourceVariable", "TabIndex", "Validation", + } +) + +// encodeElement is encodeWidget's sibling for a sub-element built directly, +// without going through the widget dispatch. +func encodeElement(t *testing.T, el element.Element) bsonv1.D { + t.Helper() + out, err := (&codec.Encoder{}).Encode(el) + if err != nil { + t.Fatalf("encode: %v", err) + } + var doc bsonv1.D + if err := bsonv1.Unmarshal([]byte(out), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + return doc +} + +func docKeys(doc bsonv1.D) []string { + out := make([]string, 0, len(doc)) + for _, e := range doc { + out = append(out, e.Key) + } + sort.Strings(out) + return out +} + +func assertKeys(t *testing.T, doc bsonv1.D, want []string) { + t.Helper() + got := docKeys(doc) + sorted := append([]string(nil), want...) + sort.Strings(sorted) + if strings.Join(got, ",") != strings.Join(sorted, ",") { + t.Errorf("keys\n got %v\n want %v", got, sorted) + } +} + +func TestStaticImageMatchesStudioProShape(t *testing.T) { + img := &pages.StaticImage{Responsive: true} + img.Name = "i1" + doc := encodeWidget(t, img) + + if got := docGet(doc, "$Type"); got != "Forms$StaticImageViewer" { + t.Fatalf("$Type = %v", got) + } + assertKeys(t, doc, studioProStaticImageKeys) + // An unset by-name reference is "", never null. Measured across ako/TestApp: + // 0 nulls against 4,400+ empty strings over 40 (type, property) pairs. + if got := docGet(doc, "Image"); got != "" { + t.Errorf("Image = %#v, want the empty string", got) + } + assertEmptyClientTemplate(t, doc, "AlternativeText") +} + +func TestDynamicImageMatchesMetamodelShape(t *testing.T) { + img := &pages.DynamicImage{Responsive: true} + img.Name = "i2" + doc := encodeWidget(t, img) + + if got := docGet(doc, "$Type"); got != "Forms$ImageViewer" { + t.Fatalf("$Type = %v", got) + } + assertKeys(t, doc, dynamicImageKeys) + if got := docGet(doc, "DefaultImage"); got != "" { + t.Errorf("DefaultImage = %#v, want the empty string", got) + } + assertEmptyClientTemplate(t, doc, "AlternativeText") + + src, ok := docGet(doc, "DataSource").(bsonv1.D) + if !ok { + t.Fatalf("DataSource = %T, want a Forms$ImageViewerSource", docGet(doc, "DataSource")) + } + if got := docGet(src, "$Type"); got != "Forms$ImageViewerSource" { + t.Errorf("DataSource.$Type = %v", got) + } +} + +func TestDropDownMatchesMetamodelShape(t *testing.T) { + dd := &pages.DropDown{} + dd.Name = "d1" + doc := encodeWidget(t, dd) + + if got := docGet(doc, "$Type"); got != "Forms$DropDown" { + t.Fatalf("$Type = %v", got) + } + assertKeys(t, doc, dropDownKeys) + // EmptyOptionCaption is the blank option's caption — a holder, not a null. + cap, ok := docGet(doc, "EmptyOptionCaption").(bsonv1.D) + if !ok { + t.Fatalf("EmptyOptionCaption = %T, want a Texts$Text", docGet(doc, "EmptyOptionCaption")) + } + if got := docGet(cap, "$Type"); got != "Texts$Text" { + t.Errorf("EmptyOptionCaption.$Type = %v", got) + } +} + +// assertEmptyClientTemplate pins the AlternativeText holder to the three Studio +// Pro references: Fallback and Template are empty Texts$Text, Parameters is an +// empty list under marker 2, and there is NO FallbackValue — that key does not +// exist on Forms$ClientTemplate (generated/metamodel: Fallback / Parameters / +// Template), and an invented key is what Studio Pro reports as "Sequence +// contains no matching element" while mxbuild builds it at 0 errors. +func assertEmptyClientTemplate(t *testing.T, parent bsonv1.D, key string) { + t.Helper() + ct, ok := docGet(parent, key).(bsonv1.D) + if !ok { + t.Fatalf("%s = %T, want a Forms$ClientTemplate", key, docGet(parent, key)) + } + if got := docGet(ct, "$Type"); got != "Forms$ClientTemplate" { + t.Errorf("%s.$Type = %v", key, got) + } + if docGet(ct, "FallbackValue") != nil { + t.Errorf("%s carries a FallbackValue; Forms$ClientTemplate has no such property", key) + } + for _, sub := range []string{"Fallback", "Template"} { + txt, ok := docGet(ct, sub).(bsonv1.D) + if !ok { + t.Errorf("%s.%s = %T, want a Texts$Text", key, sub, docGet(ct, sub)) + continue + } + if got := docGet(txt, "$Type"); got != "Texts$Text" { + t.Errorf("%s.%s.$Type = %v", key, sub, got) + } + // Marker 3 and no translations. Measured: every one of the 3,257 + // Texts$Text lists in ako/TestApp uses marker 3, empty or not — legacy + // writes 2 for a non-empty one, which is the other way this drifted. + items, ok := docGet(txt, "Items").(bsonv1.A) + if !ok || len(items) != 1 || items[0] != int32(3) { + t.Errorf("%s.%s.Items = %#v, want [3]", key, sub, docGet(txt, "Items")) + } + } + params, ok := docGet(ct, "Parameters").(bsonv1.A) + if !ok || len(params) != 1 || params[0] != int32(2) { + t.Errorf("%s.Parameters = %#v, want [2]", key, docGet(ct, "Parameters")) + } +} + +// TestNanoflowSourceNestsSettings — gen binds the nanoflow name directly on the +// source; Studio Pro nests it in a Forms$NanoflowSettings child. Writing gen's +// shape would put the name in a key Studio Pro does not read there. +func TestNanoflowSourceNestsSettings(t *testing.T) { + el := nanoflowSourceToGen(&pages.NanoflowSource{Nanoflow: "MyModule.NF_GetItems"}) + doc := encodeElement(t, el) + + if got := docGet(doc, "$Type"); got != "Forms$NanoflowSource" { + t.Fatalf("$Type = %v", got) + } + if docGet(doc, "Nanoflow") != nil { + t.Error("Nanoflow bound directly on the source; it belongs in NanoflowSettings") + } + settings, ok := docGet(doc, "NanoflowSettings").(bsonv1.D) + if !ok { + t.Fatalf("NanoflowSettings = %T", docGet(doc, "NanoflowSettings")) + } + if got := docGet(settings, "Nanoflow"); got != "MyModule.NF_GetItems" { + t.Errorf("NanoflowSettings.Nanoflow = %v", got) + } +} diff --git a/mdl/backend/modelsdk/widget_write_signout_test.go b/mdl/backend/modelsdk/widget_write_signout_test.go index a454018d1..46052fa96 100644 --- a/mdl/backend/modelsdk/widget_write_signout_test.go +++ b/mdl/backend/modelsdk/widget_write_signout_test.go @@ -64,8 +64,15 @@ func TestClientActionToGen_StillRefusesWhatItCannotWrite(t *testing.T) { if err == nil { t.Fatal("an action with no writer was accepted, which means dropping it") } - if !strings.Contains(err.Error(), "not yet supported") { - t.Errorf("unexpected message: %v", err) + // The message no longer says "not yet supported by the modelsdk engine — + // rerun with MXCLI_ENGINE=legacy": legacy cannot write this either, so that + // sent the reader to a workaround that was never going to work. What it must + // still do is name the action, so the report says which one. + if !strings.Contains(err.Error(), "ShowHomePageClientAction") { + t.Errorf("the error does not name the action: %v", err) + } + if strings.Contains(err.Error(), "MXCLI_ENGINE=legacy") { + t.Errorf("points at legacy, which cannot write it either: %v", err) } } diff --git a/mdl/executor/authoring_language_test.go b/mdl/executor/authoring_language_test.go index e2b2aa88e..5f5c4832b 100644 --- a/mdl/executor/authoring_language_test.go +++ b/mdl/executor/authoring_language_test.go @@ -92,15 +92,20 @@ func TestPageTextsUseProjectDefaultLanguage(t *testing.T) { } assertTextLang(t, "button Caption", btn.CaptionTemplate.Template, lang, "Opslaan") - // Static text content — the same builder family, a different property. - txt, err := pb.buildTextWidgetV3(&ast.WidgetV3{ - Type: "text", Name: "t", + // Dynamic text content — the same builder family, a different + // property. (It was `text` until buildTextWidgetV3 started refusing: + // that keyword wrote Forms$Text, which Mendix does not have.) + txt, err := pb.buildDynamicTextV3(&ast.WidgetV3{ + Type: "dynamictext", Name: "t", Properties: map[string]any{"Content": "Welkom"}, }) if err != nil { - t.Fatalf("buildTextWidgetV3: %v", err) + t.Fatalf("buildDynamicTextV3: %v", err) } - assertTextLang(t, "text Content", txt.Caption, lang, "Welkom") + if txt.Content == nil { + t.Fatal("dynamic text has no Content") + } + assertTextLang(t, "dynamictext Content", txt.Content.Template, lang, "Welkom") }) } } diff --git a/mdl/executor/cmd_pages_builder_dataview_test.go b/mdl/executor/cmd_pages_builder_dataview_test.go index 473ab5aef..dc6ee780f 100644 --- a/mdl/executor/cmd_pages_builder_dataview_test.go +++ b/mdl/executor/cmd_pages_builder_dataview_test.go @@ -23,7 +23,7 @@ func dataViewWith(props map[string]any, children ...*ast.WidgetV3) *ast.WidgetV3 func footerBlock() *ast.WidgetV3 { return &ast.WidgetV3{Type: "footer", Name: "f", Children: []*ast.WidgetV3{ - {Type: "text", Name: "t", Properties: map[string]any{"Content": "x"}}, + {Type: "dynamictext", Name: "t", Properties: map[string]any{"Content": "x"}}, }} } diff --git a/mdl/executor/cmd_pages_builder_v3_widgets.go b/mdl/executor/cmd_pages_builder_v3_widgets.go index 4f5df93ae..caa11e4b2 100644 --- a/mdl/executor/cmd_pages_builder_v3_widgets.go +++ b/mdl/executor/cmd_pages_builder_v3_widgets.go @@ -636,39 +636,21 @@ func (pb *pageBuilder) buildRadioButtonsV3(w *ast.WidgetV3) (*pages.RadioButtons return rb, nil } +// buildTextWidgetV3 used to build a Forms$Text. It now refuses, because Mendix +// has no such type: the written project fails to LOAD with +// TypeCacheUnknownTypeException, so `mx check` and Studio Pro both reject it +// before any validation runs. See validate_widget_retired.go for the +// measurement — `mxcli check` reports the `statictext` spelling as MDL-WIDGET27, +// and this is the backstop for `text`, which resolves through the widget +// registry and so is only caught by check when a project is available. +// +// Reading one is still supported: an old project converted up can carry a +// Forms$Text, and rewriting its page preserves it (widget_write_legacy_gaps.go). +// Only creating a new one is refused. func (pb *pageBuilder) buildTextWidgetV3(w *ast.WidgetV3) (*pages.Text, error) { - st := &pages.Text{ - BaseWidget: pages.BaseWidget{ - BaseElement: model.BaseElement{ - ID: model.ID(types.GenerateID()), - TypeName: "Forms$Text", - }, - Name: w.Name, - }, - RenderMode: pages.TextRenderModeText, - } - - // Handle Content - if content := w.GetContent(); content != "" { - st.Caption = &model.Text{ - BaseElement: model.BaseElement{ - ID: model.ID(types.GenerateID()), - TypeName: "Texts$Text", - }, - Translations: map[string]string{pb.textLang(): content}, - } - } - - // Handle RenderMode - if rm := w.GetRenderMode(); rm != "" { - st.RenderMode = pages.TextRenderMode(rm) - } - - if err := pb.registerWidgetName(w.Name, st.ID); err != nil { - return nil, err - } - - return st, nil + return nil, fmt.Errorf("widget `%s`: `%s` writes Forms$Text, a type Mendix does not have — "+ + "the project could not be opened afterwards; use `dynamictext` instead", + w.Name, strings.ToLower(w.Type)) } // dynamicTextVariableRe matches a DYNAMICTEXT Content value that is a variable diff --git a/mdl/executor/validate_widget_retired.go b/mdl/executor/validate_widget_retired.go new file mode 100644 index 000000000..39f1c370d --- /dev/null +++ b/mdl/executor/validate_widget_retired.go @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// A widget keyword whose Mendix type no longer exists. +// +// `statictext` writes `Forms$Text`, and Mendix 11 has no such type. The result +// is not a build error — it is a project that cannot be LOADED: +// +// ERROR: System.AggregateException: One or more errors occurred. +// (The type cache does not contain a type with qualified name Forms$Text.) +// ---> Mendix.Modeler.Storage.Caches.TypeCacheUnknownTypeException +// +// `mx check` fails there, before any validation runs, and Studio Pro fails the +// same way — so the page is unopenable and unfixable in the modeler. Measured on +// mxbuild 11.14.0 against ako/TestApp, and reproduced identically with +// MXCLI_ENGINE=legacy: both writers emit the type, so this is not an engine +// difference and switching engines is not a workaround. +// +// `Forms$Text` is absent from BOTH generated sources — zero occurrences in +// modelsdk/gen and no type in generated/metamodel — and from 3,257 Studio +// Pro-authored text holders in ako/TestApp. The modern widget is +// `Forms$DynamicText`, which `dynamictext` writes. +// +// The READERS keep handling `Forms$Text` (cmd_pages_describe_parse.go, +// cmd_page_wireframe.go, theme_reader.go): a project converted up from an old +// Mendix version can still carry one, and describing it is useful. Only writing +// a new one is refused. +var retiredWidgetKinds = map[string]struct { + storedType string + replacedBy string +}{ + "statictext": {storedType: "Forms$Text", replacedBy: "dynamictext"}, +} + +// validateRetiredWidgetKind refuses a widget keyword whose stored $Type Mendix +// no longer has. +// +// This is deliberately not version-gated. mxcli has no Mendix version in which +// `Forms$Text` is known — the oldest reflection snapshot in the repo +// (generated/metamodel, 11.6.0) does not declare it either — so a gate would +// have no version to let through, and guessing one would trade a clear refusal +// for an unopenable project. +func validateRetiredWidgetKind(w *ast.WidgetV3, locationPrefix string) []linter.Violation { + if w == nil || w.TypeIsGeneric { + return nil + } + r, ok := retiredWidgetKinds[strings.ToLower(w.Type)] + if !ok { + return nil + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET27", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` writes %s, a type Mendix does not have — the project cannot be opened afterwards", + locationPrefix, strings.ToLower(w.Type), r.storedType), + Suggestion: fmt.Sprintf("use `%s` instead; it writes the widget Mendix actually has", r.replacedBy), + }} +} diff --git a/mdl/executor/validate_widget_retired_test.go b/mdl/executor/validate_widget_retired_test.go new file mode 100644 index 000000000..87375f1f2 --- /dev/null +++ b/mdl/executor/validate_widget_retired_test.go @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// `statictext` writes Forms$Text, and Mendix 11 has no such type. The project +// that comes out cannot be LOADED — `mx check` and Studio Pro both stop at +// TypeCacheUnknownTypeException before any validation runs — so the page is +// unopenable and cannot be repaired in the modeler. +// +// Both engines emit it, so this is not something MXCLI_ENGINE works around; the +// refusal is in the builder and the check, not in a writer. +func TestValidateRetiredWidgetKind_StaticTextIsRefused(t *testing.T) { + // Deliberately no project: the type is unknown to every Mendix version, so + // unlike MDL-WIDGET25 this must fire without one. Passing "" is the + // assertion, not a convenience. + got := validateRetiredWidgetKind(&ast.WidgetV3{Type: "statictext", Name: "t1"}, "page X") + if len(got) == 0 || got[0].RuleID != "MDL-WIDGET27" { + t.Fatalf("statictext was accepted; got %v", got) + } + if !strings.Contains(got[0].Message, "Forms$Text") { + t.Errorf("the message does not name the type that is missing: %s", got[0].Message) + } + if !strings.Contains(got[0].Suggestion, "dynamictext") { + t.Errorf("the suggestion does not name the replacement: %s", got[0].Suggestion) + } + + // And it reaches the tree walk, or `mxcli check` never runs it. + if v := widgetKindViolations(t, "", []*ast.WidgetV3{{Type: "statictext", Name: "t1"}}); !containsRule(v, "MDL-WIDGET27") { + t.Errorf("not reported by the widget-tree walk: %v", v) + } +} + +// The control. Without it this test would pass just as well against a rule that +// refused every widget — and `dynamictext` is the exact widget the refusal above +// tells the author to use, so it is the one spelling that must stay clean. +func TestValidateRetiredWidgetKind_DynamicTextIsSilent(t *testing.T) { + got := widgetKindViolations(t, "", []*ast.WidgetV3{ + {Type: "dynamictext", Name: "t1", Properties: map[string]any{"Content": "hello"}}, + }) + if containsRule(got, "MDL-WIDGET27") { + t.Errorf("dynamictext was refused: %v", got) + } +} + +// The builder is the backstop for `text`, which shares buildTextWidgetV3 but +// reaches the validator as a generic type — so MDL-WIDGET25 only catches it when +// a project is available to resolve against. Neither spelling may reach a write. +func TestBuildTextWidget_RefusesBothSpellings(t *testing.T) { + for _, kind := range []string{"text", "statictext"} { + t.Run(kind, func(t *testing.T) { + pb := &pageBuilder{} + if _, err := pb.buildTextWidgetV3(&ast.WidgetV3{Type: kind, Name: "t1"}); err == nil { + t.Fatalf("%s built a Forms$Text widget", kind) + } else if !containsText([]string{err.Error()}, "Forms$Text") { + t.Errorf("the error does not name the type: %v", err) + } + }) + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index edbead1ba..e089facd6 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -124,6 +124,9 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // Slice 0: is this a widget at all, and does the parent declare this // container? Both were previously left to `exec`. out = append(out, validateWidgetKind(w, registry, lookupWidgetDef(parent, registry), parentObjectLists, locationPrefix)...) + // A keyword whose stored $Type Mendix no longer has. Unlike MDL-WIDGET25 + // this needs no project: the type is unknown to every Mendix version. + out = append(out, validateRetiredWidgetKind(w, locationPrefix)...) out = append(out, validatePluggableWidgetProperties(w, registry, locationPrefix)...) // A repeatable property written as a property value — `attributes: // [(…)]` — which used to check clean, exec, and vanish (#999). Runs for diff --git a/sdk/mpr/writer_widgets_display.go b/sdk/mpr/writer_widgets_display.go index b7b995bf5..85b638e55 100644 --- a/sdk/mpr/writer_widgets_display.go +++ b/sdk/mpr/writer_widgets_display.go @@ -849,12 +849,18 @@ func serializeStaticImage(img *pages.StaticImage) bson.D { doc := bson.D{ {Key: "$ID", Value: idToBsonBinary(string(img.ID))}, {Key: "$Type", Value: "Forms$StaticImageViewer"}, + // AlternativeText is not optional — generated/metamodel declares it + // without omitempty and all three Studio-Pro-authored static images in + // ako/TestApp carry it. It used to be omitted here. + {Key: "AlternativeText", Value: emptyAlternativeText()}, {Key: "Appearance", Value: serializeAppearance(img.Class, img.Style, img.DynamicClasses, img.DesignProperties)}, {Key: "ClickAction", Value: serializeClientAction(img.OnClickAction)}, {Key: "ConditionalVisibilitySettings", Value: nil}, {Key: "Height", Value: int64(img.Height)}, {Key: "HeightUnit", Value: "Auto"}, - {Key: "Image", Value: nil}, + // An unset by-name reference is "", never null: measured 0 nulls against + // 4,400+ empty strings over 40 (type, property) pairs in ako/TestApp. + {Key: "Image", Value: ""}, {Key: "Name", Value: img.Name}, {Key: "NativeAccessibilitySettings", Value: nil}, {Key: "Responsive", Value: img.Responsive}, @@ -865,21 +871,40 @@ func serializeStaticImage(img *pages.StaticImage) bson.D { return doc } +// emptyAlternativeText is the Forms$ClientTemplate an image widget carries when +// no alternative text has been set — an empty Template, an empty Fallback and no +// parameters. +// +// Pinned to the three Studio-Pro-authored Forms$StaticImageViewer widgets in +// ako/TestApp (FeedbackModule). The dynamic image used to build its own version +// of this carrying a "FallbackValue" string instead: Forms$ClientTemplate has no +// such property (generated/metamodel: Fallback / Parameters / Template), and an +// invented key is the failure Studio Pro reports as "Sequence contains no +// matching element" while mxbuild builds it at 0 errors. Note the empty +// Parameters list takes marker 2, not the 3 an empty Texts$Text takes. +func emptyAlternativeText() bson.D { + emptyText := func() bson.D { + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(generateUUID())}, + {Key: "$Type", Value: "Texts$Text"}, + {Key: "Items", Value: bson.A{int32(3)}}, + } + } + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(generateUUID())}, + {Key: "$Type", Value: "Forms$ClientTemplate"}, + {Key: "Fallback", Value: emptyText()}, + {Key: "Parameters", Value: bson.A{int32(2)}}, + {Key: "Template", Value: emptyText()}, + } +} + // serializeDynamicImage serializes a DynamicImage widget. func serializeDynamicImage(img *pages.DynamicImage) bson.D { doc := bson.D{ {Key: "$ID", Value: idToBsonBinary(string(img.ID))}, {Key: "$Type", Value: "Forms$ImageViewer"}, - {Key: "AlternativeText", Value: bson.D{ - {Key: "$ID", Value: idToBsonBinary(generateUUID())}, - {Key: "$Type", Value: "Forms$ClientTemplate"}, - {Key: "FallbackValue", Value: ""}, - {Key: "Template", Value: bson.D{ - {Key: "$ID", Value: idToBsonBinary(generateUUID())}, - {Key: "$Type", Value: "Texts$Text"}, - {Key: "Items", Value: bson.A{int32(3)}}, - }}, - }}, + {Key: "AlternativeText", Value: emptyAlternativeText()}, {Key: "Appearance", Value: serializeAppearance(img.Class, img.Style, img.DynamicClasses, img.DesignProperties)}, {Key: "ClickAction", Value: serializeClientAction(img.OnClickAction)}, {Key: "ConditionalVisibilitySettings", Value: nil}, @@ -888,7 +913,8 @@ func serializeDynamicImage(img *pages.DynamicImage) bson.D { {Key: "$Type", Value: "Forms$ImageViewerSource"}, {Key: "EntityRef", Value: nil}, }}, - {Key: "DefaultImage", Value: nil}, + // "" not null — an unset by-name reference; see serializeStaticImage. + {Key: "DefaultImage", Value: ""}, {Key: "Height", Value: int64(img.Height)}, {Key: "HeightUnit", Value: "Auto"}, {Key: "Name", Value: img.Name}, diff --git a/sdk/mpr/writer_widgets_image_test.go b/sdk/mpr/writer_widgets_image_test.go new file mode 100644 index 000000000..3403bc078 --- /dev/null +++ b/sdk/mpr/writer_widgets_image_test.go @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The image widgets, pinned to the shape Mendix stores. +// +// This is the legacy half of the pair; the modelsdk half is +// mdl/backend/modelsdk/widget_write_legacy_gaps_test.go and asserts the same +// things about the same widgets. Keeping both is the point: the two engines had +// silently drifted apart here, and only one of them was right. +// +// Ground truth is the three Studio-Pro-authored Forms$StaticImageViewer widgets +// ako/TestApp inherits from FeedbackModule, plus generated/metamodel (the +// arbiter per CLAUDE.md) for Forms$ImageViewer, which no reference project in +// reach carries. + +package mpr + +import ( + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/sdk/pages" + + "go.mongodb.org/mongo-driver/bson" +) + +func imgKeys(doc bson.D) string { + out := make([]string, 0, len(doc)) + for _, e := range doc { + out = append(out, e.Key) + } + sort.Strings(out) + return strings.Join(out, ",") +} + +// TestStaticImageMatchesStudioPro — legacy used to omit AlternativeText, which +// generated/metamodel declares without omitempty and all three references carry, +// and to write BSON null for the unset Image. An unset by-name reference is the +// empty string: measured 0 nulls against 4,400+ empty strings over 40 +// (type, property) pairs in ako/TestApp. +func TestStaticImageMatchesStudioPro(t *testing.T) { + img := &pages.StaticImage{Responsive: true} + img.Name = "i1" + doc := serializeStaticImage(img) + + want := "$ID,$Type,AlternativeText,Appearance,ClickAction," + + "ConditionalVisibilitySettings,Height,HeightUnit,Image,Name," + + "NativeAccessibilitySettings,Responsive,TabIndex,Width,WidthUnit" + if got := imgKeys(doc); got != want { + t.Errorf("keys\n got %s\n want %s", got, want) + } + if got := bsonLookup(doc, "Image"); got != "" { + t.Errorf("Image = %#v, want the empty string", got) + } + assertEmptyClientTemplateBSON(t, doc, "AlternativeText") +} + +// TestDynamicImageMatchesMetamodel — legacy's AlternativeText here was +// hand-rolled and carried a FallbackValue key that Forms$ClientTemplate does not +// have, four lines after a comment in the shared serializer saying exactly that +// ("Must be Fallback object, not FallbackValue string"). +func TestDynamicImageMatchesMetamodel(t *testing.T) { + img := &pages.DynamicImage{Responsive: true} + img.Name = "i2" + doc := serializeDynamicImage(img) + + want := "$ID,$Type,AlternativeText,Appearance,ClickAction," + + "ConditionalVisibilitySettings,DataSource,DefaultImage,Height,HeightUnit," + + "Name,NativeAccessibilitySettings,OnClickEnlarge,Responsive," + + "ShowAsThumbnail,TabIndex,Width,WidthUnit" + if got := imgKeys(doc); got != want { + t.Errorf("keys\n got %s\n want %s", got, want) + } + if got := bsonLookup(doc, "DefaultImage"); got != "" { + t.Errorf("DefaultImage = %#v, want the empty string", got) + } + assertEmptyClientTemplateBSON(t, doc, "AlternativeText") +} + +func assertEmptyClientTemplateBSON(t *testing.T, parent bson.D, key string) { + t.Helper() + ct := bsonSubDoc(t, parent, key) + if got := bsonLookup(ct, "$Type"); got != "Forms$ClientTemplate" { + t.Errorf("%s.$Type = %v", key, got) + } + if bsonLookup(ct, "FallbackValue") != nil { + t.Errorf("%s carries a FallbackValue; Forms$ClientTemplate has no such property "+ + "(metamodel: Fallback / Parameters / Template). Studio Pro refuses to open a "+ + "document with an unknown property; mxbuild builds it at 0 errors", key) + } + for _, sub := range []string{"Fallback", "Template"} { + txt := bsonSubDoc(t, ct, sub) + if got := bsonLookup(txt, "$Type"); got != "Texts$Text" { + t.Errorf("%s.%s.$Type = %v", key, sub, got) + } + items, ok := bsonLookup(txt, "Items").(bson.A) + if !ok || len(items) != 1 || items[0] != int32(3) { + t.Errorf("%s.%s.Items = %#v, want [3]", key, sub, bsonLookup(txt, "Items")) + } + } + params, ok := bsonLookup(ct, "Parameters").(bson.A) + if !ok || len(params) != 1 || params[0] != int32(2) { + t.Errorf("%s.Parameters = %#v, want [2]", key, bsonLookup(ct, "Parameters")) + } +} From a139d3888827cad59595d31e1dc1d47f95de4909 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 09:34:51 +0000 Subject: [PATCH 10/18] feat(modelsdk): implement the only two reachable unimplemented backend methods MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nineteen of FullBackend's 276 methods were left to the errUnimplemented stub, whose message tells the user to rerun with MXCLI_ENGINE=legacy — the engine being retired. That reads as nineteen reasons legacy has to stay shipped. Measured, it is two. Grep cannot settle this: `b.reader.GetRawUnitByName(...)` inside the MPR backend and `ctx.Backend.GetRawUnitByName(...)` in the executor look identical to a regex and receiver names vary. scripts/backend-reachability.sh asks the compiler instead — remove one method from its interface, rebuild, and see whether anything fails. 17 DEAD, 2 LIVE. GetRawUnitByName and ParseMicroflowBSON are called from four sites in mdl/executor/cmd_microflows_builder.go (lookupMicroflowReturnType, lookupNanoflowReturnType, microflowExists, nanoflowExists). Nothing was broken by their absence and no test failed, because each is a fast path with a working fallback: the stub errored and the code fell through to an O(n) walk of the module's microflows, producing the same answer. That is why this went unnoticed, and it is the shape to look for when auditing what an engine cannot do. Both are now implemented. GetRawUnitByName and its sibling GetRawMicroflowByName are one-line delegations — the codec reader already indexes by name. ParseMicroflowBSON needed real work: the callers pass NANOFLOW contents to it as well, which the legacy parser only survived because it walks an untyped bson map, while the codec decodes to two different gen types. No speedup to report. Restoring the fast path measured 620ms against 608ms over three runs on a 138-microflow project — within noise, because `mxcli exec` runs `check` first and check's own helpers load ListMicroflows anyway, so the cache the slow path builds is already warm. The value is the removal of the errUnimplemented cliff on the default engine. The 17 dead methods are left in place rather than purged: they are surface, not dead code paths — implemented and working on the MPR backend — and deleting them touches six files for no behavioural change. TestNoReachableUnimplementedBackendMethods pins the probe's output instead, with the reason each was found unreachable, so a new stub or a dropped implementation fails there and prompts a re-run of the script rather than an extension of the list on faith. Its control asserts the two live methods stay implemented, or the test would pass just as well against a build where nothing is. Controls: removing either implementation makes its test fail with "not implemented yet — rerun with MXCLI_ENGINE=legacy"; un-implementing a listed-dead method makes the guard name it and print the script to run; a stale entry in the list is reported too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + mdl/backend/modelsdk/raw_lookup.go | 96 +++++++++++ mdl/backend/modelsdk/raw_lookup_test.go | 151 ++++++++++++++++++ .../unimplemented_reachability_test.go | 123 ++++++++++++++ scripts/backend-reachability.sh | 59 +++++++ 5 files changed, 430 insertions(+) create mode 100644 mdl/backend/modelsdk/raw_lookup.go create mode 100644 mdl/backend/modelsdk/raw_lookup_test.go create mode 100644 mdl/backend/modelsdk/unimplemented_reachability_test.go create mode 100755 scripts/backend-reachability.sh diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index 0466f8d82..85cf95459 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -92,3 +92,4 @@ {"area": "mdl/backend", "date": "2026-09-11", "symptom": "Making SOAP calls describe structurally instead of as base64 silently rewrote Studio Pro's own documents: a describe -> exec round trip over ako/TestApp flipped `Range.SingleObject` false -> true with no error at all, and turned Clients.SaveOrder's `DataTypes$BooleanType` result into VoidType, which mxbuild reported as `[CE0366]` + `[CE6011]`", "cause": "`webServiceActionRequiresRawBSON` admitted keys BY NAME. That was sound while the supported set was the only nine keys the writer emitted \u2014 but a real call carries FIFTEEN, and six of the new ones are boilerplate mxcli writes at ONE fixed value. Admitting `HttpConfiguration`, `RequestHeaderHandling`, `IsValidationRequired`, `ProxyConfiguration`, `RequestProxyType` and `NewResultHandling` by name meant any call configured beyond mxcli's defaults would be normalised on the next exec", "file": "`mdl/backend/modelsdk/microflow_read_actions.go`, `sdk/mpr/parser_microflow_actions.go` (webServiceActionRequiresRawBSON and its value predicates)", "insight": "**The question a raw-fallback gate answers is not 'do I know this key' but 'would writing this back produce the same document'.** The two coincide only while the writer emits exactly the supported set; the moment a feature lands that widens what is representable, the by-name test starts approving documents it cannot reproduce \u2014 and the loss is invisible, because the result is a VALID model that differs from the user's. **The round trip is the only thing that catches it**: unit tests on the new feature all passed, `mx check` on the newly-written calls was 0 errors, and the regression only appeared when describe -> exec was run over the REFERENCE documents and the BSON diffed (`mxcli bson dump --type microflow --object`, ids normalised away). Two of the five diffs that surfaced were pre-existing microflow describe drift (NoCase case values, bezier control vectors) and unrelated \u2014 worth separating before blaming the change. Fixing it also SHRANK the feature's reach honestly: Studio Pro's own SOAP calls keep the raw form until `Range.SingleObject` is explained, and only mxcli-authored calls describe structurally. **A result type can come from somewhere MDL cannot see** \u2014 SaveOrder's Boolean is the WSDL operation's return type, not an import mapping's entity \u2014 so 'binds a result' does not imply 'derivable'", "refs": []} {"area": "mdl/backend", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec` \u2014 the documented copy operation \u2014 silently turns OFF a microflow's \"apply entity access\". Measured across 342 microflows in 4 projects (11.14.0): every microflow storing `ApplyEntityAccess: true` came back `false` (16/342, all 4 distinct Administration documents). A blocking `show message` also becomes non-blocking (16 microflows). `mxcli check` and mxbuild are both silent \u2014 the model is valid either way", "cause": "TWO causes wearing one symptom. (1) `ApplyEntityAccess` is HARDCODED false in both writers (`mdl/backend/modelsdk/microflow_write.go` SetApplyEntityAccess(false), `sdk/mpr/writer_microflow.go` {Key:\"ApplyEntityAccess\", Value:false}) and `microflows.Microflow` has no field for it, so the read side drops it first. (2) `ShowMessageAction.Blocking` is carried correctly end to end on BOTH engines \u2014 the loss is in DESCRIBE, which has no `blocking` keyword to emit, so the re-parse sets false", "file": "`mdl/backend/modelsdk/microflow_write.go`, `sdk/mpr/writer_microflow.go`, `sdk/microflows/microflows.go` (Microflow struct), `mdl/executor/cmd_microflows_format_action.go` (show message)", "insight": "**A round-trip audit needs a preservation control, or it cannot tell a bug from its own blind spot.** Here it was `StableId`: 342/342 preserved, exactly as ADR-0008 claims \u2014 a method that normalised ids away too aggressively would have reported that as churn, and everything else with it. **Separate `hardcoded in the writer` from `unspellable in DESCRIBE`**: they look identical in a before/after diff and need completely different fixes (model plumbing vs grammar), and `Blocking` proves a property can be perfectly carried by both engines and still be lost by the text round trip. **The same property handled two ways in one codebase is the tell**: `microflows.Rule` carries ApplyEntityAccess correctly (`rule_write.go`, `parser_rule.go`) while `microflows.Microflow` does not \u2014 so this was never an unknown-property gap, just an unfinished one. **Check what an 'empty' value actually holds before calling it a loss**: `ConcurrenyErrorMessage` looked like 58 lost translations and every single one had `Text: \"\"`, i.e. an empty entry versus no entry. Two more traps worth knowing: `ActionActivity.Caption` changes ONLY where `AutoGenerateCaption` is true (a user-set caption survives), and ~90% of the 417-line diff on a real microflow is layout \u2014 bezier vectors, sizes, connection indices \u2014 which buries the two lines that matter", "refs": []} {"area": "mdl/backend", "date": "2026-09-12", "symptom": "Four plain widget keywords were refused by the DEFAULT engine with \"widget *pages.X not yet supported by the modelsdk engine — rerun with MXCLI_ENGINE=legacy\": statictext, dropdown, staticimage, dynamicimage (plus NanoflowSource as a list data source). Nothing in mdl-examples/ or in any skill used those keywords, so the doctype gate ran green the whole time. Then, closing the gap: `statictext` on EITHER engine produced a project that could not be loaded at all — `mx check` aborts with TypeCacheUnknownTypeException for Forms$Text before running any validation, and Studio Pro fails the same way", "cause": "Two unrelated things. (1) The modelsdk widget dispatch simply had no case arms for those five types — reachable set measured at five, not the twenty-three a name scan suggests: nineteen pages.* widget structs are constructed by nothing, and EntityPathSource / ShowHomePageClientAction are written by NEITHER engine, so their fallback named a path legacy could not take either. (2) `Forms$Text` is not a Mendix type. It is absent from modelsdk/gen, from generated/metamodel, and from all 3,257 Studio Pro Texts$Text holders in ako/TestApp; `buildTextWidgetV3` minted it for both `text` and `statictext`. Writing it is worse than a build error — the project is unopenable and unfixable in the modeler", "file": "`mdl/backend/modelsdk/widget_write_legacy_gaps.go` (new), `mdl/backend/modelsdk/widget_write.go`, `sdk/mpr/writer_widgets_display.go`, `mdl/executor/validate_widget_retired.go` (new), `mdl/executor/cmd_pages_builder_v3_widgets.go`", "insight": "**\"Parity with the other engine\" is the wrong bar until you have checked whether the other engine is right.** That was the starting assumption here and measuring overturned it: ako/TestApp carries three Studio-Pro-authored Forms$StaticImageViewer widgets (FeedbackModule) and legacy disagreed with all three — it omitted AlternativeText (non-omitempty in generated/metamodel) and wrote BSON null for the unset Image. Legacy's dynamic image was worse: a hand-rolled AlternativeText carrying a FallbackValue key that Forms$ClientTemplate does not have, four lines after a comment in the SHARED serializer saying exactly that (\"Must be Fallback object, not FallbackValue string\"). sdk/mpr was corrected to match rather than pinned as truth. **Look for a Studio Pro reference inside the fixture before concluding there is none** — `bson.Unmarshal` every mprcontents unit and count instances of the $Type; the marketplace modules a blank app ships are Studio Pro output. The same 40-line scanner settles value-shape questions no doc answers: measured 0 nulls against 4,400+ empty strings for by-name references, so an unset QualifiedName is \"\", never null; and marker 3 in 3,257 of 3,257 Texts$Text lists, so legacy's marker 2 for a non-empty one is the wrong side of that divergence, not modelsdk's 3. **A widget keyword the example corpus never uses is a keyword nothing tests**, whatever the coverage numbers say. **Check that the project still LOADS, not just that it builds** — a load failure aborts `mx check` before the error list, so grepping for CE-codes finds nothing and reads like success", "refs": []} +{"area": "mdl/backend", "date": "2026-09-12", "symptom": "The modelsdk (default) engine leaves 19 of FullBackend's 276 methods to the errUnimplemented stub, which tells the user to \"rerun with MXCLI_ENGINE=legacy\" — the engine being retired. Nineteen unported methods reads as nineteen reasons legacy has to stay shipped and tested", "cause": "Seventeen of the nineteen cannot be reached at all: they are interface surface that only the MPR backend's own delegation and callers holding a concrete *mpr.Reader / *mpr.Writer (api/, examples/, cmd/mxcli commands that open a reader directly) ever touch, so the stub can never fire. The two that ARE reachable — GetRawUnitByName and ParseMicroflowBSON, four call sites in mdl/executor/cmd_microflows_builder.go — were invisible because each is a fast path with a working slow-path fallback: the stub errored, the code fell through to an O(n) module walk, and the result was identical", "file": "`mdl/backend/modelsdk/raw_lookup.go` (new), `mdl/backend/modelsdk/unimplemented_reachability_test.go` (new), `scripts/backend-reachability.sh` (new)", "insight": "**Grep cannot answer \"is this interface method reachable\" and the compiler can.** `b.reader.GetRawUnitByName(...)` inside the MPR backend and `ctx.Backend.GetRawUnitByName(...)` in the executor are indistinguishable to a regex, and receiver names vary, so a grep-based count said all nineteen had callers. Deleting one method from its interface at a time and rebuilding gives a yes/no per method with no judgement involved: 17 DEAD, 2 LIVE. Slow (one `go build ./...` each, minutes for the set) but decisive — script it, commit it, and pin its OUTPUT in a test rather than re-running it in CI. **A fallback hides an unimplemented method completely.** Nothing was broken and no test failed, because the fast path's failure was indistinguishable from a cache miss; the only symptom was work being done twice. Look for `if x, err := …; err == nil` fast paths when auditing what an engine cannot do. **Measure the speedup before claiming one**: restoring the fast path made no measurable difference (620ms vs 608ms over three runs on a 138-microflow project, within noise), because `mxcli exec` runs `check` first and check's own helpers load ListMicroflows anyway — so the cache the slow path builds is already warm. The value here is the removal of the errUnimplemented cliff, not speed", "refs": []} diff --git a/mdl/backend/modelsdk/raw_lookup.go b/mdl/backend/modelsdk/raw_lookup.go new file mode 100644 index 000000000..ac228b992 --- /dev/null +++ b/mdl/backend/modelsdk/raw_lookup.go @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + bsonv2 "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + genMf "github.com/mendixlabs/mxcli/modelsdk/gen/microflows" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// The only two FullBackend methods the modelsdk engine did not implement that +// anything actually calls through the interface. +// +// Measured with scripts/backend-reachability.sh, which removes one method from +// its interface at a time and rebuilds: 17 of the 19 unimplemented methods have +// no caller through a backend value at all, and these two have four, all in +// mdl/executor/cmd_microflows_builder.go (lookupMicroflowReturnType, +// lookupNanoflowReturnType, microflowExists, nanoflowExists). +// +// Nothing was broken by their absence, which is why this went unnoticed: each of +// the four is a fast path with a slow-path fallback, so on the default engine +// the fast path errored with "not implemented yet" and the code fell through to +// an O(n) walk that loads and parses every microflow in the module. The cost was +// speed, not correctness — and the ONLY remaining reason a default-engine run +// could hit errUnimplemented, which is what stands between here and dropping +// legacy from nightly. + +// GetRawUnitByName returns a unit's raw info by object type and qualified name. +// The codec reader already indexes by name; this was simply never exposed as a +// Backend method (the same gap units.go describes for GetRawUnitBytes). +func (b *Backend) GetRawUnitByName(objectType, qualifiedName string) (*types.RawUnitInfo, error) { + return b.reader.GetRawUnitByName(objectType, qualifiedName) +} + +// GetRawMicroflowByName returns a microflow unit's raw BSON by qualified name. +// +// Unreachable through the interface today — the reachability probe found no +// caller — but it is the sibling of the method above on the same interface, the +// reader already has it, and leaving exactly one of a pair stubbed is the shape +// that produces a puzzling failure later. +func (b *Backend) GetRawMicroflowByName(qualifiedName string) ([]byte, error) { + return b.reader.GetRawMicroflowByName(qualifiedName) +} + +// ParseMicroflowBSON decodes a stored microflow or nanoflow document. +// +// Both, despite the name: the callers pass nanoflow contents to it as well and +// read ReturnType off the result. The legacy parser is untyped — it walks a +// bson map, so a nanoflow document parses into a Microflow struct by accident of +// sharing key names. The codec is typed and decodes to two different gen types, +// so the nanoflow case has to be handled deliberately rather than falling out. +// +// containerID is passed through rather than resolved: the callers already know +// which module they asked about, and a lookup here would undo the point of the +// fast path. +func (b *Backend) ParseMicroflowBSON(contents []byte, unitID, containerID model.ID) (*microflows.Microflow, error) { + elem, err := codec.NewDecoder(codec.DefaultRegistry).Decode(bsonv2.Raw(contents)) + if err != nil { + return nil, err + } + var out *microflows.Microflow + switch g := elem.(type) { + case *genMf.Microflow: + out = microflowFromGen(g, containerID) + case *genMf.Nanoflow: + // The interface returns a Microflow, so a nanoflow is carried in one. + // Only the fields the callers read are meaningful here; a nanoflow is + // not a microflow and this value must not be written back. + nf := nanoflowFromGen(g, containerID) + if nf == nil { + return nil, nil + } + out = µflows.Microflow{ + ContainerID: nf.ContainerID, + Name: nf.Name, + Documentation: nf.Documentation, + Excluded: nf.Excluded, + AllowedModuleRoles: nf.AllowedModuleRoles, + ReturnType: nf.ReturnType, + Parameters: nf.Parameters, + ObjectCollection: nf.ObjectCollection, + } + out.ID = nf.ID + out.TypeName = "Microflows$Nanoflow" + default: + return nil, nil + } + if out != nil && unitID != "" { + out.ID = unitID + } + return out, nil +} diff --git a/mdl/backend/modelsdk/raw_lookup_test.go b/mdl/backend/modelsdk/raw_lookup_test.go new file mode 100644 index 000000000..7fa819e98 --- /dev/null +++ b/mdl/backend/modelsdk/raw_lookup_test.go @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func connectedBackend(t *testing.T) *Backend { + t.Helper() + b := New() + if err := b.Connect(copyFixture(t)); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + return b +} + +// The name-indexed lookup the executor's fast paths use. Before this existed the +// modelsdk engine answered errUnimplemented and every one of those four call +// sites fell through to an O(n) walk of the module's microflows. +func TestGetRawUnitByName(t *testing.T) { + b := connectedBackend(t) + + unit, err := b.GetRawUnitByName("microflow", "FeedbackModule.ConvertBase64String") + if err != nil { + t.Fatalf("GetRawUnitByName: %v", err) + } + if unit == nil { + t.Fatal("microflow not found by name") + } + if len(unit.Contents) == 0 { + t.Error("unit found but carries no contents; the fast path reads Contents") + } + + // A name that is not there must not resolve — the callers treat a hit as + // proof the microflow exists (microflowExists), so a lenient lookup would + // silence a real "no such microflow" report. + if got, _ := b.GetRawUnitByName("microflow", "FeedbackModule.NoSuchMicroflow"); got != nil { + t.Errorf("a missing microflow resolved to %+v", got) + } +} + +// ParseMicroflowBSON is asked about nanoflows too, and the codec decodes those +// to a different gen type — the legacy parser only coped because it walked an +// untyped map. ReturnType is the field every caller reads. +func TestParseMicroflowBSON(t *testing.T) { + b := connectedBackend(t) + + // Note that a Void microflow still carries a DataType — "no return value" is + // a type, not a nil. Asserting on presence would have passed against a parse + // that produced nothing, so these compare the type name. + for _, tc := range []struct { + objectType string + name string + wantType string + }{ + {"microflow", "FeedbackModule.ConvertBase64String", "String"}, + {"microflow", "Administration.SaveNewAccount", "Void"}, + } { + t.Run(tc.name, func(t *testing.T) { + unit, err := b.GetRawUnitByName(tc.objectType, tc.name) + if err != nil || unit == nil { + t.Fatalf("GetRawUnitByName(%s): %v", tc.name, err) + } + mf, err := b.ParseMicroflowBSON(unit.Contents, "", "") + if err != nil { + t.Fatalf("ParseMicroflowBSON: %v", err) + } + if mf == nil { + t.Fatal("parsed to nil") + } + if mf.ReturnType == nil { + t.Fatalf("no ReturnType, want %s", tc.wantType) + } + if got := mf.ReturnType.GetTypeName(); got != tc.wantType { + t.Errorf("ReturnType = %s, want %s", got, tc.wantType) + } + // The name proves the document was really decoded rather than a + // zero value returned — a nil ReturnType alone is indistinguishable + // from a parse that produced nothing. + if mf.Name == "" { + t.Error("no Name on the parsed microflow") + } + }) + } + + if _, err := b.ParseMicroflowBSON([]byte("not bson"), "", ""); err == nil { + t.Error("malformed BSON parsed without error") + } +} + +// The fast path is what the four executor call sites use; this is the property +// that makes replacing the slow path safe — both must agree on the return type. +func TestFastPathReturnTypeMatchesTheSlowPath(t *testing.T) { + b := connectedBackend(t) + + all, err := b.ListMicroflows() + if err != nil { + t.Fatalf("ListMicroflows: %v", err) + } + byName := map[string]*microflows.Microflow{} + for _, mf := range all { + if mf != nil { + byName[mf.Name] = mf + } + } + if len(byName) == 0 { + t.Fatal("no microflows in the fixture; the comparison would be vacuous") + } + + checked := 0 + for _, qualified := range []string{ + "FeedbackModule.ConvertBase64String", + "FeedbackModule.ConvertUUIDToURL", + "Administration.SaveNewAccount", + "Administration.ChangePassword", + } { + unit, err := b.GetRawUnitByName("microflow", qualified) + if err != nil || unit == nil { + t.Errorf("%s: not found by name", qualified) + continue + } + fast, err := b.ParseMicroflowBSON(unit.Contents, "", "") + if err != nil || fast == nil { + t.Errorf("%s: parse failed: %v", qualified, err) + continue + } + slow := byName[fast.Name] + if slow == nil { + t.Errorf("%s: the slow path does not list it", qualified) + continue + } + fastType, slowType := "", "" + if fast.ReturnType != nil { + fastType = fast.ReturnType.GetTypeName() + } + if slow.ReturnType != nil { + slowType = slow.ReturnType.GetTypeName() + } + if fastType != slowType { + t.Errorf("%s: fast path says %q, slow path says %q", qualified, fastType, slowType) + } + checked++ + } + if checked == 0 { + t.Fatal("compared nothing") + } +} diff --git a/mdl/backend/modelsdk/unimplemented_reachability_test.go b/mdl/backend/modelsdk/unimplemented_reachability_test.go new file mode 100644 index 000000000..b92e67ea6 --- /dev/null +++ b/mdl/backend/modelsdk/unimplemented_reachability_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Can a default-engine run still hit errUnimplemented? +// +// That is the question standing between the modelsdk engine and dropping +// `legacy` from nightly, and counting unimplemented methods answers it wrongly: +// 19 of FullBackend's 276 methods were not declared on *Backend, but 17 of them +// are interface surface nothing calls through a backend value, so their stub +// could never fire. Measured with scripts/backend-reachability.sh, which removes +// one method from its interface at a time and rebuilds — grep cannot tell +// `b.reader.X()` inside the MPR backend from `ctx.Backend.X()` in the executor. +// +// This test does not repeat that measurement (a build per method takes minutes). +// It pins its OUTPUT: the set of methods *Backend leaves to the stub must be +// exactly the set measured unreachable. A new stub, or a rename that drops an +// implementation, fails here and is a prompt to re-run the script rather than to +// extend the list on faith. +package modelsdkbackend + +import ( + "reflect" + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// unreachableUnimplemented are the FullBackend methods *Backend does not +// implement, each measured to have no caller through a backend value. +// +// The reason column is what the probe found, not a guess: for all but one it is +// that only the MPR backend's own delegation mentions the name, plus callers +// that hold a concrete *mpr.Reader / *mpr.Writer (the api/ package, examples/, +// and cmd/mxcli commands that open a reader directly) — none of which route +// through this engine. +var unreachableUnimplemented = map[string]string{ + "AddAttribute": "api/ and examples/ call it on the sdk writer; ALTER ENTITY goes through the mutator", + "UpdateAttribute": "same as AddAttribute", + "GetDomainModelByID": "the MPR and MCP backends call it on their own reader", + "ExportJSON": "examples/read_project calls it on the sdk reader", + "FindCustomWidgetType": "cmd/mxcli/cmd_extract_templates.go holds a concrete reader", + "FindAllCustomWidgetTypes": "reached only via the reader, inside modelsdk/mpr itself", + "GetProjectRootID": "callers hold a reader; this package uses b.reader.GetProjectRootID directly", + "GetUnitTypes": "no caller at all outside the MPR delegation", + "ListAllUnitIDs": "cmd/mxcli/diag.go holds a concrete reader; infrastructure_write.go uses b.reader", + "ListRawUnits": "the bson dump/discover/describe commands hold a concrete reader", + "ListNavigationDocuments": "the MCP backend and sdk/mpr call it on their own reader", + "GetWorkflow": "the MCP backend calls it on its own reader", + "UpdateLayout": "ALTER LAYOUT goes through the page mutator, not this method", + "SerializeWidget": "the child serializer is a separate type (codecChildSerializer), not Backend", + "SerializeClientAction": "same as SerializeWidget", + "SerializeDataSource": "no caller at all, on any type", +} + +func TestNoReachableUnimplementedBackendMethods(t *testing.T) { + declared := methodsDeclaredOnBackend(t) + iface := reflect.TypeOf((*backend.FullBackend)(nil)).Elem() + + var missing []string + for i := 0; i < iface.NumMethod(); i++ { + if name := iface.Method(i).Name; !declared[name] { + missing = append(missing, name) + } + } + if len(missing) == 0 && len(unreachableUnimplemented) > 0 { + t.Fatalf("*Backend now implements everything, but %d methods are still listed as "+ + "unreachable — delete unreachableUnimplemented rather than leave it stale", + len(unreachableUnimplemented)) + } + sort.Strings(missing) + + var unexpected []string + for _, name := range missing { + if unreachableUnimplemented[name] == "" { + unexpected = append(unexpected, name) + } + } + if len(unexpected) > 0 { + t.Errorf("these FullBackend methods fall through to the errUnimplemented stub and are not\n"+ + "recorded as unreachable:\n %s\n"+ + "Run `scripts/backend-reachability.sh %s`. If it says LIVE, implement the method —\n"+ + "a default-engine run can reach it and will be told to rerun on an engine that is\n"+ + "being retired. If it says DEAD, add it to unreachableUnimplemented with that reason.", + strings.Join(unexpected, "\n "), strings.Join(unexpected, " ")) + } + + present := map[string]bool{} + for _, name := range missing { + present[name] = true + } + var stale []string + for name := range unreachableUnimplemented { + if !present[name] { + stale = append(stale, name) + } + } + sort.Strings(stale) + if len(stale) > 0 { + t.Errorf("unreachableUnimplemented lists %v, which *Backend now implements — "+ + "strike them off, or the list stops meaning anything", stale) + } +} + +// TestTheTwoReachableMethodsAreImplemented is the control for the list above. +// +// Without it the test would pass just as well against a build where nothing is +// implemented and everything is listed as unreachable. These two are the ones +// the probe found LIVE, called from mdl/executor/cmd_microflows_builder.go, and +// they must never go back on the list. +func TestTheTwoReachableMethodsAreImplemented(t *testing.T) { + declared := methodsDeclaredOnBackend(t) + for _, name := range []string{"GetRawUnitByName", "ParseMicroflowBSON"} { + if !declared[name] { + t.Errorf("%s is reachable from the executor but not implemented — a default-engine "+ + "run hits errUnimplemented and falls back to an O(n) module walk", name) + } + if unreachableUnimplemented[name] != "" { + t.Errorf("%s is listed as unreachable; the probe found four call sites in "+ + "mdl/executor/cmd_microflows_builder.go", name) + } + } +} diff --git a/scripts/backend-reachability.sh b/scripts/backend-reachability.sh new file mode 100755 index 000000000..64bc1988e --- /dev/null +++ b/scripts/backend-reachability.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Which backend.FullBackend methods does anything actually CALL through a +# backend value? +# +# The modelsdk engine answers unported methods with errUnimplemented ("rerun +# with MXCLI_ENGINE=legacy"), which is a reason the legacy engine has to stay +# shipped and tested. Counting the unimplemented ones overstates the problem +# badly: most are interface surface that only the MPR implementation and its own +# reader ever touch, so no engine can reach them and the stub can never fire. +# +# Grep cannot tell the difference — `b.reader.GetRawUnitByName(...)` inside the +# MPR backend and `ctx.Backend.GetRawUnitByName(...)` in the executor look the +# same and receiver names vary. The compiler can: remove one method from its +# interface and rebuild. A clean build means nothing calls it through a backend +# value. +# +# Measured on 2026-09-12 over the 19 methods *Backend did not declare: 17 DEAD, +# 2 LIVE (GetRawUnitByName, ParseMicroflowBSON — four call sites, all in +# mdl/executor/cmd_microflows_builder.go). Both are now implemented; +# TestNoReachableUnimplementedBackendMethods in mdl/backend/modelsdk holds the +# list this script produced. +# +# Slow on purpose: one full `go build ./...` per method, a few minutes for the +# whole set. Run it when the unimplemented list changes, not routinely. +# +# Usage: +# scripts/backend-reachability.sh # every method *Backend lacks +# scripts/backend-reachability.sh GetWorkflow … # just these + +set -u +cd "$(dirname "$0")/.." || exit 1 + +methods=("$@") +if [ ${#methods[@]} -eq 0 ]; then + echo "usage: $0 [Method...]" >&2 + echo "(the current list is in mdl/backend/modelsdk/unimplemented_reachability_test.go)" >&2 + exit 2 +fi + +trap 'git checkout -- mdl/backend/ 2>/dev/null' EXIT + +for m in "${methods[@]}"; do + # Where is it declared? Interface methods sit at one tab of indent. + f=$(grep -rl "^ $m(" mdl/backend/*.go | head -1) + if [ -z "$f" ]; then + echo "SKIP: $m (no interface declaration found)" + continue + fi + cp "$f" /tmp/backend-reachability.bak + perl -i -pe "s{^(\t$m\()}{\t// PROBE \$1}" "$f" + out=$(go build ./... 2>&1 | grep -v '^#' | head -4) + cp /tmp/backend-reachability.bak "$f" + if [ -z "$out" ]; then + echo "DEAD: $m" + else + echo "LIVE: $m" + echo "$out" | sed 's/^/ /' + fi +done From d4bb899f0842d54d50189f96b8608767a4b03547 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 09:48:49 +0000 Subject: [PATCH 11/18] fix(check): renumber the statictext rule, and stop legacy running on every Mendix version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things, both settled by measurement. **MDL-WIDGET27 was already taken.** The `statictext` refusal added in 141da93f reused an ID that validate_widget_object_property.go owns — and MDL-WIDGET28, the next number, is taken too by a validator that is on main but not on this branch, which is 18 commits behind. It is now MDL-WIDGET29. A rule ID is allocated globally and was being picked against a local view; `git grep` on the working tree would have caught the first collision and could never have caught the second. TestRuleIDHasOneOwner guards the half a repository can see: an ID raised from two different validator files fails, with the reminder to check the default branch rather than the working tree. The invariant is one OWNER, not one occurrence — MDL-WIDGET25 legitimately fires from two branches of its own validator, so counting occurrences would be a false-alarm generator. Adding it immediately found a second case, MDL059, reused earlier the same day for document-level annotations; that one is a genuine extension of one rule to a second site (someone suppressing MDL059 means both), so it is recorded on the exemption list with that reason instead of renumbered. MDL-WIDGET21's pre-existing overlap is recorded the same way: renumbering a shipped rule changes output, json/sarif and suppressions, so an existing overlap is documented rather than churned. **Legacy now runs on one Mendix version in nightly, not five.** The engine matrix is most of what that step costs — each script is executed and handed to mxbuild once per engine (measured 588s -> 232s for modelsdk alone) — so legacy was five sixths of the fleet's cost for a path nothing routes to any more: the last four widget keywords are written by the codec engine (141da93f) and the last two reachable unimplemented backend methods are implemented (a139d388). It is strictly weaker now, refusing rules, menus, layouts, message definitions and regular expressions. Narrowed rather than dropped, deliberately. `--engine legacy` is still something a user can type, and an engine that ships untested rots silently — the failure this repo has already had once (#808, an integration test that had only ever skipped). One version keeps it honest at a fifth of the cost; that line goes when the engine does. Legacy's unit tests are untouched and still run on every push. Evidence it still passes: the last eight nightlies are green, the most recent this morning, each running the full engine matrix across all five versions. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .claude/skills/mendix/custom-widgets/SKILL.md | 2 +- .github/workflows/nightly.yml | 49 ++++++-- cmd/mxcli/syntax/features_page.go | 2 +- docs-site/src/language/widget-types.md | 2 +- .../widgets-statictext-unknown-type.fail.mdl | 4 +- .../modelsdk/widget_write_legacy_gaps.go | 2 +- mdl/executor/cmd_pages_builder_v3_widgets.go | 2 +- mdl/executor/roundtrip_doctype_test.go | 7 +- mdl/executor/rule_id_uniqueness_test.go | 106 ++++++++++++++++++ mdl/executor/validate_widget_retired.go | 2 +- mdl/executor/validate_widget_retired_test.go | 6 +- 12 files changed, 164 insertions(+), 21 deletions(-) create mode 100644 mdl/executor/rule_id_uniqueness_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 79372745c..0bf1f3064 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -590,3 +590,4 @@ {"area": "mdl/executor", "date": "2026-09-11", "symptom": "`send mapping X` on a SOAP call parsed, `exec` reported success, and the mapping name appeared ZERO times in the written document \u2014 on both engines. mxbuild then rejected the call as `[CE0369] \"Cannot use simple request body, as the operation's body is complex\"`. Separately, an operation taking parameters had no MDL syntax at all and built as `[CE0178] \"Body parameter mapping needs to be refreshed.\"`", "cause": "Both writers emitted an unconditional empty `Microflows$SimpleRequestHandling` for `RequestBodyHandling`. It is a POLYMORPHIC child holding EITHER the operation's arguments (SimpleRequestHandling + WebServiceOperationSimpleParameterMapping entries) or an export mapping (`Microflows$MappingRequestHandling`, NOT the `Mendix$AdvancedRequestHandling` legacy's comment named \u2014 that type is in none of the three ako/TestApp reference documents). The reader compounded it by looking for `RequestHandling`/`ExportMappingCall`, keys no real document carries; the stored key is `RequestBodyHandling`", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go`, `sdk/mpr/writer_microflow_actions.go` (webServiceRequestBody), `mdl/executor/webservice_names.go` (webServiceParameterPath), `mdl/grammar/domains/MDLMicroflow.g4`", "insight": "**Two gaps that look independent can be one polymorphic property, and finding that out is what makes the validation possible.** Arguments and the send mapping are the two branches of `RequestBodyHandling`, so the mutual-exclusion rule (MDL-SOAP01) only becomes enforceable once BOTH exist \u2014 implementing either alone means `check` can refuse a combination it cannot offer an alternative to. **The stored ParameterPath is derivable, which is what makes readable syntax possible**: `escape(operation.RequestBodyElementName) + \"|\" + name` reads the element off `Description.Services[].Operations[]` of the imported service, so MDL says `OrderId` and not `http%3A//www.example.com/:GetOrder|OrderId`. Escaping is per SEGMENT \u2014 `:` inside a segment becomes %3A, the separator `:` and the `/`es are left alone \u2014 and a segment containing `%` is REFUSED because Mendix's escaping of it is unmeasured. **The typed-array marker is the silent trap**: a populated ParameterMappings list leads with marker 2 and `codec.lookupListMarker` DEFAULTS TO 3, so without a `RegisterListMarker` the arguments serialize under the wrong array version \u2014 invisible to mxbuild, fatal to Studio Pro. Control: stubbing the registration makes the test report `marker = 3, want int32(2)`", "refs": []} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "`describe microflow` -> `exec`, the documented copy operation, silently turned OFF a microflow's \"apply entity access\" (16/342 microflows across 4 projects, 11.14.0) and turned a blocking `show message` into a non-blocking one (16 microflows). Neither `mxcli check` nor mxbuild says anything: the model is valid either way, and the only consequence is that a constrained user behaves differently", "cause": "TWO causes, one symptom. ApplyEntityAccess was HARDCODED false in both writers and `microflows.Microflow` had no field, so the read side dropped it first \u2014 while `microflows.Rule` carried it correctly in the same codebase (and its CREATE path never set it, so rules had the same bug from the other end). ShowMessageAction.Blocking was carried perfectly by BOTH engines; MDL simply had no keyword, so DESCRIBE could not emit it and the re-parse set false", "file": "`sdk/microflows/microflows.go`, `mdl/backend/modelsdk/microflow.go`+`microflow_write.go`, `sdk/mpr/parser_microflow.go`+`writer_microflow.go`, `mdl/executor/apply_entity_access.go`, `mdl/executor/cmd_microflows_build.go`, `mdl/executor/cmd_rules_create.go`, `mdl/grammar/MDLLexer.g4` (BLOCKING)", "insight": "**Carrying a property through the writers is only half a round-trip fix when the middle is MDL text.** describe -> exec rebuilds from the STATEMENT, so a setting the statement never names is gone however well the storage layer handles it \u2014 which is why ApplyEntityAccess needed BOTH preserve-on-rewrite (for CREATE OR REPLACE) and an annotation (for the copy case, where there is nothing to preserve from). **Absent must not mean false for a security setting**: the AST field is a *bool so `@applyentityaccess(false)` and silence are distinguishable, the same rule @excluded (#914) and the doc comment (#1018) already follow. No grammar change was needed \u2014 `annotationValue` already accepts a literal, so a parameterised annotation is free. **Adding a lexer keyword needs a keyword-rule entry and a test**: BLOCKING would otherwise stop `blocking` being usable as a parameter name. **The fix is only believable with the audit re-run as its verification** \u2014 three unit-test controls plus the same 342-microflow measurement showing both properties stopped moving. That re-run also surfaced CE0709 \"Sequence flow is not accepted by origin or destination\" on a whole-project round trip, which the pre-fix binary reproduces identically: a separate, pre-existing flow-graph defect that would have been easy to misattribute to this change", "refs": []} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "An annotation written before a CREATE that the document does not read \u2014 a typo (`@applyentityacces`), the right name on the wrong document kind (`@applyentityaccess` on a nanoflow), or an activity annotation at document level (`@caption`) \u2014 passed `mxcli check`, passed `exec`, and built at 0 errors with the annotation silently dropped", "cause": "MDL's grammar attaches `annotation*` to `createStatement` ITSELF, so all forty-odd create kinds accept an annotation while only six read one. Each document's builder looks only for the names it implements, so a name NO builder implements is invisible to all of them \u2014 there was nowhere the check could have lived. MDL059 already refused exactly this one node family over (on statements), and #884's reasoning applies unchanged", "file": "`mdl/visitor/visitor_document_annotations.go` (ExitCreateStatement), `mdl/executor/validate_document_annotations.go`, `mdl/ast/ast.go` (Program.DocumentAnnotations)", "insight": "**When a check has no natural home in any single handler, the parse tree is the home.** Deriving the document kind from the grammar's OWN rule names (`parser.MDLParserParserStaticData.RuleNames`, `createMicroflowStatement` -> \"microflow\") beats forty type assertions and means a create statement added later is covered the day it is added, defaulting to 'reads no annotations' rather than being silently forgotten. **Record everything, decide centrally**: the visitor logs every document annotation with its kind and the validator owns the policy, so the accepted set is one table beside `knownActivityAnnotations` instead of being spread across the seven visitor sites. **Pin the table to the visitor in BOTH directions** \u2014 a name the visitor reads but the table omits rejects a valid script; a name the table lists but nothing reads re-opens the hole. The AST-scraping test that does this needs one non-obvious filter: match only EqualFold calls whose other operand mentions `AnnotationName`, or it also collects the literal \"false\" from an annotation that compares its own VALUE. **Measure the blast radius before erroring on something previously silent**: a scan of every .mdl and skill block found exactly the seven implemented combinations and no stray annotation, and both MDL corpora plus the full suite stayed green \u2014 without that, turning silence into an error is a guess", "refs": []} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "Two unrelated validators shipped the same rule ID. `statictext` was given MDL-WIDGET27, which validate_widget_object_property.go already owned (a repeatable widget property written as a value) — and MDL-WIDGET28, the next number, was taken too, by a validator that exists on the default branch but not on the working branch", "cause": "The branch was 18 commits behind main. `git grep MDL-WIDGET27` on the WORKING TREE would have found the first collision and was not run; nothing at all would have found the second, because that rule is not in the branch. Rule IDs are allocated by picking the next unused number, which is a global allocation done against a local view", "file": "`mdl/executor/validate_widget_retired.go`, `mdl/executor/rule_id_uniqueness_test.go` (new)", "insight": "**A rule ID is allocated against the DEFAULT BRANCH, not the working tree** — `git grep origin/main` before claiming a number, and `git grep -ho 'MDL0[0-9][0-9]' origin/main | sort -u | tail` to find the real high-water mark. A branch behind by a few commits sees a free number that is not. **The invariant to test is one OWNER, not one occurrence**: a rule legitimately fires from several branches of its own validator (MDL-WIDGET25 does), so counting occurrences is a false alarm generator; counting distinct FILES is the useful proxy. Adding that guard immediately found a second collision from earlier the same day — MDL059 reused for a document-level annotation — which on inspection was a genuine extension of one rule to a second site (someone suppressing MDL059 means both), so it went on the exemption list with that reason rather than being renumbered. **Renumbering a shipped rule is user-visible** (it appears in output, --format json/sarif, and suppressions), which is why an existing overlap gets recorded rather than fixed. The guard cannot see a rule that is not in the repository, and says so in its own comment — that limit is the other half of the same bug", "refs": []} diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index f45ff400f..8b6b42d57 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -81,7 +81,7 @@ near-miss suggestions), and a container the parent does not declare is MDL-WIDGET26. Both need `-p`: without a project, mxcli knows only its embedded widgets, so it stays quiet rather than reporting every real widget as unknown. -MDL-WIDGET27 needs no project: `statictext` writes `Forms$Text`, a type Mendix +MDL-WIDGET29 needs no project: `statictext` writes `Forms$Text`, a type Mendix does not have, and the project that comes out cannot be *loaded* at all (`mx check` and Studio Pro both stop at `TypeCacheUnknownTypeException` before validation). Use `dynamictext` with a literal `Content:`. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index ccf696e23..a683c27ec 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -14,7 +14,20 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - mendix-version: ['10.24.24.119349', '11.6.8', '11.12.2', '11.13.0', '11.14.0'] + # `engines` is spelled out per entry rather than defaulted, so reading + # this block tells you where legacy runs without also knowing what an + # omitted key falls back to. + include: + - mendix-version: '10.24.24.119349' + engines: modelsdk + - mendix-version: '11.6.8' + engines: modelsdk + - mendix-version: '11.12.2' + engines: modelsdk + - mendix-version: '11.13.0' + engines: modelsdk + - mendix-version: '11.14.0' + engines: all fail-fast: false name: test (Mendix ${{ matrix.mendix-version }}) steps: @@ -44,14 +57,36 @@ jobs: - name: Setup mxbuild ${{ matrix.mendix-version }} run: ./bin/mxcli setup mxbuild --version ${{ matrix.mendix-version }} - # The full engine matrix runs here, not on every push. "all" is the - # default, so this is belt and braces — but it is the line that makes the - # nightly the place legacy is verified, and it should survive someone - # changing the default. - - name: "Integration tests (Mendix ${{ matrix.mendix-version }}, engines: all)" + # The nightly is where legacy is verified — the per-push job runs modelsdk + # alone (push-test.yml) — but it no longer runs on every Mendix version. + # + # The engine matrix is most of what this step costs: each script is + # executed and handed to mxbuild once PER ENGINE (measured 588s -> 232s + # for modelsdk alone), so legacy was five sixths of the fleet's cost for a + # path nothing routes to any more. As of 2026-09-12 no mxcli feature falls + # back to it: the last four widget keywords are written by the codec engine + # and the last two reachable unimplemented backend methods are implemented + # (mdl/backend/modelsdk/unimplemented_reachability_test.go holds that + # measurement). Legacy is now strictly weaker — rules, menus, layouts, + # message definitions and regular expressions all refuse on it. + # + # It is narrowed rather than dropped, deliberately. `--engine legacy` is + # still a thing a user can type, and an engine that ships untested rots + # silently — the failure this repo has already had once (#808, an + # integration test that had only ever skipped). One version keeps the + # engine honest at a fifth of the cost; drop this line only together with + # the engine itself. + # + # Legacy's UNIT tests are unaffected and still run on every push: what + # narrows here is only the expensive exec + mxbuild loop. + # + # The engine set is in the step NAME as well as the env, because `go test` + # without -v discards a passing package's output, so TestMain's own notice + # never reaches a green log. + - name: "Integration tests (Mendix ${{ matrix.mendix-version }}, engines: ${{ matrix.engines }})" run: make test-integration env: - MXCLI_TEST_ENGINES: all + MXCLI_TEST_ENGINES: ${{ matrix.engines }} timeout-minutes: 30 nightly: diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 6ba4de889..7334f9d72 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -145,7 +145,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "-- DROPDOWN -> COMBOBOX\n" + "-- And three the executor refuses on BOTH engines, each with its own message:\n" + "-- STATICTEXT (writes Forms$Text, a type Mendix no longer has — the\n" + - "-- project could not be OPENED afterwards; MDL-WIDGET27.\n" + + "-- project could not be OPENED afterwards; MDL-WIDGET29.\n" + "-- Use DYNAMICTEXT with a literal Content.)\n" + "-- REFERENCESELECTOR (unsupported widget type)\n" + "-- LEGACYDATAGRID (use DATAGRID for the pluggable equivalent on Mendix 11+)", diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index ce184227f..c4b94783e 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -508,7 +508,7 @@ accepted widget — MDL-WIDGET25, with the nearest known names suggested. A container keyword the parent widget does not declare is MDL-WIDGET26. Both need a project open (`-p`), since without one mxcli knows only its embedded widgets. -A third, MDL-WIDGET27, needs no project: `statictext` writes `Forms$Text`, and +A third, MDL-WIDGET29, needs no project: `statictext` writes `Forms$Text`, and Mendix has no such type. That is not a build error but a **load** error — `mx check` and Studio Pro both stop at `TypeCacheUnknownTypeException` before any validation runs, so the page cannot even be opened to repair it. Use diff --git a/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl b/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl index abc8f7525..f9a180abf 100644 --- a/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl +++ b/mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl @@ -1,7 +1,7 @@ --- `statictext` writes a type Mendix does not have (refusal, MDL-WIDGET27). +-- `statictext` writes a type Mendix does not have (refusal, MDL-WIDGET29). -- -- `mxcli check mdl-examples/bug-tests/widgets-statictext-unknown-type.fail.mdl` --- reports MDL-WIDGET27. Before the fix this file checked clean, executed, and +-- reports MDL-WIDGET29. Before the fix this file checked clean, executed, and -- produced a project that could not be LOADED at all: -- -- ERROR: System.AggregateException: One or more errors occurred. diff --git a/mdl/backend/modelsdk/widget_write_legacy_gaps.go b/mdl/backend/modelsdk/widget_write_legacy_gaps.go index 314f7387c..328a5d8de 100644 --- a/mdl/backend/modelsdk/widget_write_legacy_gaps.go +++ b/mdl/backend/modelsdk/widget_write_legacy_gaps.go @@ -35,7 +35,7 @@ import ( // The first of the five turned out not to be a gap at all. Closing it and // running the result through `mx check` showed that BOTH engines wrote a project // that could not be LOADED — Mendix has no Forms$Text — so `statictext` is now -// refused at build and check time (MDL-WIDGET27, +// refused at build and check time (MDL-WIDGET29, // mdl/executor/validate_widget_retired.go). Nothing constructs pages.Text any // more, so there is no writer for it here either — an old project that carries // one keeps it because ALTER PAGE mutates the stored gen document rather than diff --git a/mdl/executor/cmd_pages_builder_v3_widgets.go b/mdl/executor/cmd_pages_builder_v3_widgets.go index caa11e4b2..1939ba161 100644 --- a/mdl/executor/cmd_pages_builder_v3_widgets.go +++ b/mdl/executor/cmd_pages_builder_v3_widgets.go @@ -640,7 +640,7 @@ func (pb *pageBuilder) buildRadioButtonsV3(w *ast.WidgetV3) (*pages.RadioButtons // has no such type: the written project fails to LOAD with // TypeCacheUnknownTypeException, so `mx check` and Studio Pro both reject it // before any validation runs. See validate_widget_retired.go for the -// measurement — `mxcli check` reports the `statictext` spelling as MDL-WIDGET27, +// measurement — `mxcli check` reports the `statictext` spelling as MDL-WIDGET29, // and this is the backstop for `text`, which resolves through the widget // registry and so is only caught by check when a project is available. // diff --git a/mdl/executor/roundtrip_doctype_test.go b/mdl/executor/roundtrip_doctype_test.go index d096e5897..4ec306ade 100644 --- a/mdl/executor/roundtrip_doctype_test.go +++ b/mdl/executor/roundtrip_doctype_test.go @@ -45,9 +45,10 @@ var allGateEngines = []gateEngine{ // It exists because the matrix is most of what the gate costs: each script is // executed and then handed to mxbuild once PER ENGINE, and mxbuild dominates. // On CI the per-push job runs `modelsdk` alone (see .github/workflows/ -// push-test.yml) and the nightly runs the full matrix across the Mendix-version -// matrix (nightly.yml), so legacy stays verified daily without every push -// paying for it. +// push-test.yml) and the nightly runs both — but on ONE Mendix version rather +// than all five (nightly.yml), since nothing routes to legacy any more and it +// was five sixths of that fleet's cost. Legacy stays verified daily, at a fifth +// of what it used to cost, and its unit tests still run on every push. // // The DEFAULT is every engine, deliberately. Nightly could have relied on a // default of "modelsdk" and set "all" itself, but then a mistake in EITHER diff --git a/mdl/executor/rule_id_uniqueness_test.go b/mdl/executor/rule_id_uniqueness_test.go new file mode 100644 index 000000000..266c666eb --- /dev/null +++ b/mdl/executor/rule_id_uniqueness_test.go @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" +) + +// One rule ID, one rule. +// +// A rule may be raised from several places in ITS OWN validator — MDL-WIDGET25 +// fires from two branches of validate_widget_kind.go — so the invariant is not +// "one occurrence" but "one owner". Two validators sharing an ID give two +// unrelated errors the same name, which breaks the thing rule IDs exist for: +// looking one up, suppressing it, or matching it in a test. +// +// This exists because it happened. A new rule was numbered MDL-WIDGET27 while +// validate_widget_object_property.go already owned that number — the branch it +// was written on had not been updated in 18 commits, so `git grep` on the branch +// found the collision and the author did not run it. +// +// Its limit, which is the other half of that story: the branch was also behind +// MDL-WIDGET28, added upstream and not present locally at all. No test in a +// repository can see a rule that is not in it. Before claiming a NEW number, +// grep the default branch, not just the working tree. +var ruleIDPattern = regexp.MustCompile(`RuleID:\s+"([A-Z][A-Z0-9-]*)"`) + +// ruleIDsSharedDeliberately are IDs raised by more than one validator today, +// each with the reason it has not been split. Renumbering a shipped rule is a +// user-visible change (it appears in output, in --format json and sarif, and in +// suppressions), so an existing overlap is recorded rather than fixed here. +var ruleIDsSharedDeliberately = map[string]string{ + "MDL-WIDGET21": "validate_widget_contentparams.go and validate_widget_editability.go " + + "both report a property the widget does not honour; predates this test", + "MDL059": "one rule, two sites: an annotation that parses and does nothing. " + + "validate_flow_parameters.go covers one written on a PARAMETER, " + + "validate_document_annotations.go one written before a CREATE. Someone " + + "suppressing MDL059 means both, so splitting the number would be wrong", +} + +func TestRuleIDHasOneOwner(t *testing.T) { + owners := map[string]map[string]bool{} + + for _, dir := range []string{".", "../linter", "../linter/rules"} { + entries, err := os.ReadDir(dir) + if err != nil { + continue + } + for _, e := range entries { + name := e.Name() + if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + data, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + for _, m := range ruleIDPattern.FindAllStringSubmatch(string(data), -1) { + if owners[m[1]] == nil { + owners[m[1]] = map[string]bool{} + } + owners[m[1]][filepath.Join(dir, name)] = true + } + } + } + if len(owners) == 0 { + t.Fatal("found no rule IDs; this test no longer measures anything") + } + + var clashes []string + for id, files := range owners { + if len(files) < 2 || ruleIDsSharedDeliberately[id] != "" { + continue + } + var names []string + for f := range files { + names = append(names, f) + } + sort.Strings(names) + clashes = append(clashes, id+" — "+strings.Join(names, ", ")) + } + sort.Strings(clashes) + if len(clashes) > 0 { + t.Errorf("these rule IDs are raised by more than one validator:\n %s\n"+ + "Give the newer rule the next free number — and check the DEFAULT BRANCH for it, "+ + "not just this working tree.", strings.Join(clashes, "\n ")) + } + + // The exemption list must not outlive its reason, or it quietly re-opens the + // hole it documents. + var stale []string + for id := range ruleIDsSharedDeliberately { + if len(owners[id]) < 2 { + stale = append(stale, id) + } + } + sort.Strings(stale) + if len(stale) > 0 { + t.Errorf("ruleIDsSharedDeliberately lists %v, which no longer clash — strike them off", stale) + } +} diff --git a/mdl/executor/validate_widget_retired.go b/mdl/executor/validate_widget_retired.go index 39f1c370d..8377e1e65 100644 --- a/mdl/executor/validate_widget_retired.go +++ b/mdl/executor/validate_widget_retired.go @@ -58,7 +58,7 @@ func validateRetiredWidgetKind(w *ast.WidgetV3, locationPrefix string) []linter. return nil } return []linter.Violation{{ - RuleID: "MDL-WIDGET27", + RuleID: "MDL-WIDGET29", Severity: linter.SeverityError, Message: fmt.Sprintf("%s: `%s` writes %s, a type Mendix does not have — the project cannot be opened afterwards", locationPrefix, strings.ToLower(w.Type), r.storedType), diff --git a/mdl/executor/validate_widget_retired_test.go b/mdl/executor/validate_widget_retired_test.go index 87375f1f2..3d439090e 100644 --- a/mdl/executor/validate_widget_retired_test.go +++ b/mdl/executor/validate_widget_retired_test.go @@ -21,7 +21,7 @@ func TestValidateRetiredWidgetKind_StaticTextIsRefused(t *testing.T) { // unlike MDL-WIDGET25 this must fire without one. Passing "" is the // assertion, not a convenience. got := validateRetiredWidgetKind(&ast.WidgetV3{Type: "statictext", Name: "t1"}, "page X") - if len(got) == 0 || got[0].RuleID != "MDL-WIDGET27" { + if len(got) == 0 || got[0].RuleID != "MDL-WIDGET29" { t.Fatalf("statictext was accepted; got %v", got) } if !strings.Contains(got[0].Message, "Forms$Text") { @@ -32,7 +32,7 @@ func TestValidateRetiredWidgetKind_StaticTextIsRefused(t *testing.T) { } // And it reaches the tree walk, or `mxcli check` never runs it. - if v := widgetKindViolations(t, "", []*ast.WidgetV3{{Type: "statictext", Name: "t1"}}); !containsRule(v, "MDL-WIDGET27") { + if v := widgetKindViolations(t, "", []*ast.WidgetV3{{Type: "statictext", Name: "t1"}}); !containsRule(v, "MDL-WIDGET29") { t.Errorf("not reported by the widget-tree walk: %v", v) } } @@ -44,7 +44,7 @@ func TestValidateRetiredWidgetKind_DynamicTextIsSilent(t *testing.T) { got := widgetKindViolations(t, "", []*ast.WidgetV3{ {Type: "dynamictext", Name: "t1", Properties: map[string]any{"Content": "hello"}}, }) - if containsRule(got, "MDL-WIDGET27") { + if containsRule(got, "MDL-WIDGET29") { t.Errorf("dynamictext was refused: %v", got) } } From f75aa4a664b4fed2ad81676c5e18802d0a3fb50d Mon Sep 17 00:00:00 2001 From: Andrej Koelewijn Date: Sat, 12 Sep 2026 18:29:36 +0000 Subject: [PATCH 12/18] docs(wiki): re-synthesize check-mxbuild-drift and widget-type-object-drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 90 of the 123 findings undigested since the 2026-08-31 bug-pattern sync are mdl/executor, and most are one of these two classes. check-mxbuild-drift gains three sections: - Three answers, not two. A check needs resolved / not-resolved / "could not establish", where exec needs two — turning "I could not look" into "your attribute is missing" is a false error that blocks a script which builds cleanly. Four findings measured that false positive independently. Includes that check normally runs with NO project, which is also the CI default, and the allow-list/deny-list choice for rules keyed on a type. - Don't model what you can run. The mutator-probe technique: run the real setter against a throwaway deep copy whose Save is refused, keep only the error, so check and exec cannot drift. - The instruments lie in specific ways. One error per microflow hides the one being measured; a load failure suppresses the error-count line while still exiting non-zero (third occurrence); the corpus sweep has a noise floor and is blind to a misparse, because it diffs diagnostics rather than the AST. Plus a remedy beat: two rules shipped whose Fix: line walked a working project into a broken one. widget-type-object-drift was the oldest and thinnest page in the category and lacked the framing that makes CE0463 tractable: that two unrelated bugs wear the error (package upgraded after authoring, versus authored-fresh-and-wrong) and the two controls separating them; that `mxcli docker check` runs mx update-widgets first and so reports 0 errors on a project that has one; that a difference from the reference is not a cause. sources: adds diagnose-ce0463.md (written after the last sync) and replaces the bare findings directory with the four shards that carry the records. Also normalises the `area` field on two findings records that used the shard filename (mdl-executor) where the other 677 use the path (mdl/executor). check-findings accepts either — it requires the field, not a vocabulary — so digest-status had bucketed both into its "areas < 5" row, which is the report used to choose this run's scope. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017s476QkXr9CFMvKspVzcvu --- .../fix-issue/findings/mdl-backend.jsonl | 2 +- .../fix-issue/findings/mdl-executor.jsonl | 2 +- docs-wiki/SYNC_LOG.md | 3 + docs-wiki/bug-patterns/check-mxbuild-drift.md | 133 ++++++++++++++---- .../bug-patterns/widget-type-object-drift.md | 98 +++++++++++-- 5 files changed, 198 insertions(+), 40 deletions(-) diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index c68238dae..1e85a5050 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -87,6 +87,6 @@ {"area": "mdl/backend", "date": "2026-09-08", "symptom": "`alter page P { set PageSize = 10 on }` errors `pluggable property \"PageSize\" not found` on a grid that `create page \u2026 (PageSize: 20)` had just written and the app really pages at. `mxcli check --references` passes the script, so it fails only at exec, after earlier statements have landed; DESCRIBE PAGE prints the same capitalised `PageSize:`, so describe \u2192 edit \u2192 exec produced a script mxcli refused to run", "cause": "A pluggable property key is lowerCamel in the widget template (`pageSize`). CREATE resolves the author's spelling case-INsensitively (widget engine `lookupProperty`, and `WidgetV3.GetStringProp` before it); ALTER went through `setPluggableWidgetPropertyMut`, which compared the template key byte-for-byte, so only the exact `pageSize` worked. Direct sequel to Findings #1 (2026-07-27), which fixed the same class for FIRST-CLASS props and deliberately left the pluggable fallback case-sensitive with the comment 'template keys must match the template exactly'", "file": "`mdl/backend/pagemutator/mutator.go` (`setPluggableWidgetPropertyMut`)", "insight": "`strings.EqualFold` against the widget's own PropertyTypes. **The disproven belief is the reusable part**: keys are STORED case-sensitively, which is not a reason to MATCH them that way \u2014 the resolver searches one object type's PropertyTypes, and across every shipped template and definition that scope holds no two keys differing only in case (96 scopes, 1208 keys, 0 collisions). Measure the ambiguity before assuming it; here there was none, and the assumption cost a whole verb. **Cheapest localiser**: run the failing statement with the template's own casing \u2014 `set pageSize` succeeded where `set PageSize` failed, in ONE measurement, on the same widget in the same project. Both engines share `pagemutator`, so an engine split says nothing here (verified: modelsdk and legacy both fixed by the one change). Tests `TestSetPluggableProperty_MatchesTemplateKeyRegardlessOfCase` (+ typo-still-errors control, + `TestPluggablePropertyKeysAreUniqueIgnoringCase` pinning the no-collision argument); repro `mdl-examples/bug-tests/alter-page-pluggable-property-casing.mdl`; verified 0 errors on `mx check` 11.13.0. Two reporter claims did NOT hold: `describe page` DOES emit PageSize, but only when it differs from the widget default 20 (deliberate, so describe round-trips) \u2014 at the default it is omitted, which reads as 'describe cannot show it'. Still open and separate: `check --references` does not resolve pluggable property names at all, so a genuine typo (`PagSize`) still checks clean and fails at exec", "refs": ["mendixlabs/mxcli#1069"]} {"area": "mdl/backend", "date": "2026-09-09", "symptom": "A control that should have proven a guard was load-bearing PASSED with the guard removed — the test could not distinguish a correct writer from one that reset the property on every write", "cause": "The fixture stored `false` for a boolean property, and false is also the zero value. A writer that ignored the spec entirely wrote false; the correct writer preserved false. Identical output, so the assertion held either way", "file": "`mdl/backend/modelsdk/navigation_throw_sync_test.go`", "insight": "For a BOOLEAN property, a preservation test must exercise BOTH stored values — the non-zero one is the only case that can fail. This is the second time in one feature: CompatibilityMode needed a synthetic `true` because all seven reference configs carry false. The generalisation: when every real document agrees on a value, the fixture drawn from real documents cannot test preservation, and a synthetic counter-case is not optional. The tell is a control that fails to fail — if stubbing the guard leaves the suite green, the test is measuring nothing, and that is worse than no test because it reads as coverage", "refs": ["ako/mxcli#413", "ThrowPartialSyncError"]} {"area": "mdl/backend", "date": "2026-09-09", "symptom": "`call web service` (legacy SOAP) on the DEFAULT engine: `mxcli exec 06b-soap-examples.mdl` reported success on all three microflows and `mx check` (11.13.0) then failed the project with `[CE0008] \"No action defined.\" at Action activity 'Activity'` plus two `[CE0109] \"Undefined variable 'Root'.\"`. Only reachable by rerunning with MXCLI_ENGINE=legacy, which is why legacy was still the documented fallback", "cause": "The codec engine READ the action (`actionFromGen` \u2192 `*microflows.WebServiceCallAction`, with a raw fallback) but `microflowActionToGen` had no case for it, so it hit `default: return nil` and the enclosing ActionActivity serialized with no action at all. The #850 shape: a missing WRITE case is not an unsupported feature, it is a silent drop that exec reports as success. The CE0109s are knock-on \u2014 the dropped action never bound $Root", "file": "`mdl/backend/modelsdk/microflow_webservice_write.go` (new), `microflow_write.go` (switch case)", "insight": "**Mirror the legacy serializer, and prove it by diffing the two engines' documents \u2014 do not re-derive the shape from the metamodel.** There is no Studio Pro-authored SOAP document in this repo, so legacy's output is the only reference that exists and is also what users' projects already contain. Method: exec the same script on each engine, `mxcli bson dump` both, normalise the random $IDs, diff. Three discrepancies fell out that no amount of reading would have shown, all in how the codec emits NULLS and MARKERS: (1) a Part property with no child encodes to nil and the encoder then SKIPS the key (`if val != nil`), so an unset part is an ABSENT key, never a null one \u2014 carry the null as a primitive `bson.Null{}` value instead, which marshals in place; (2) `codec.TypeDefaults.NullFields` does emit the key but APPENDS it after every property, so it cannot reproduce alphabetical key order; (3) both NullFields and list markers are registered per `$Type` and several types are SHARED between writers \u2014 `Microflows$HttpConfiguration` needs HttpHeaderEntries marker 3 for SOAP and 2 for REST, and legacy writes CustomLocationTemplate as null for SOAP but omits it for REST, so a global registration would have silently changed the REST path. Write those explicitly per call site. (The package already carries one such collision: `Microflows$HttpHeaderEntry` is registered 2 in microflow_write.go and 3 in odata_write.go, decided by file order.) Verified: the two engines' CallWebServiceAction documents are now identical key-for-key and value-for-value; `mx check` goes 3 structural errors \u2192 0, leaving only the script's 4 deliberate CE1613 dangling refs, which is exactly legacy's output. Revert control: removing the switch case fails all six unit tests with the CE0008 message. The doctype engineScriptSkip for modelsdk/06b was removed and the script now runs on BOTH engines", "refs": []} -{"area": "mdl-backend", "date": "2026-09-10", "symptom": "`mxcli diff-local` on an MPR v2 project fails with `Error: mprcontents directory not found` while mprcontents/ exists and is populated; `MXCLI_ENGINE=legacy` works", "cause": "The modelsdk engine (the default) never overrode `Backend.ContentsDir()`, so it fell through to the generated `unimplemented` stub and returned \"\". diff-local reads \"\" as 'not a v2 project'.", "file": "mdl/backend/modelsdk/backend.go", "insight": "gen_unimplemented.go's promise that an unoverridden method 'fails loudly rather than silently dropping data' is CONDITIONAL on the method having an error to fail through: the generated body is `errUnimplemented` only when a result is `error`, a panic when there are no results at all, and a silent `var r0 T; return r0` otherwise. ContentsDir is in the third bucket and its zero value is a MEANINGFUL in-band answer (\"\" == MPR v1), so the missing implementation was indistinguishable from a v1 project rather than looking like a bug. The detectable signature was the contradiction between two questions the same command asks: Version() (implemented) says 2, ContentsDir() says v1. Guard added in mdl/backend/modelsdk/unimplemented_silent_test.go \u2014 reflect over FullBackend for error-less methods, go/parser the package for methods actually declared on *Backend, since reflection cannot tell a promoted method from an override (Go synthesises a wrapper named (*Backend).X for both). It immediately found a second one, InvalidateCache (a latent panic, no caller today).", "refs": ["mendixlabs/mxcli#1080"]} +{"area": "mdl/backend", "date": "2026-09-10", "symptom": "`mxcli diff-local` on an MPR v2 project fails with `Error: mprcontents directory not found` while mprcontents/ exists and is populated; `MXCLI_ENGINE=legacy` works", "cause": "The modelsdk engine (the default) never overrode `Backend.ContentsDir()`, so it fell through to the generated `unimplemented` stub and returned \"\". diff-local reads \"\" as 'not a v2 project'.", "file": "mdl/backend/modelsdk/backend.go", "insight": "gen_unimplemented.go's promise that an unoverridden method 'fails loudly rather than silently dropping data' is CONDITIONAL on the method having an error to fail through: the generated body is `errUnimplemented` only when a result is `error`, a panic when there are no results at all, and a silent `var r0 T; return r0` otherwise. ContentsDir is in the third bucket and its zero value is a MEANINGFUL in-band answer (\"\" == MPR v1), so the missing implementation was indistinguishable from a v1 project rather than looking like a bug. The detectable signature was the contradiction between two questions the same command asks: Version() (implemented) says 2, ContentsDir() says v1. Guard added in mdl/backend/modelsdk/unimplemented_silent_test.go \u2014 reflect over FullBackend for error-less methods, go/parser the package for methods actually declared on *Backend, since reflection cannot tell a promoted method from an override (Go synthesises a wrapper named (*Backend).X for both). It immediately found a second one, InvalidateCache (a latent panic, no caller today).", "refs": ["mendixlabs/mxcli#1080"]} {"area": "mdl/backend", "date": "2026-09-10", "symptom": "SOAP `call web service` writes a document mxbuild accepts and Studio Pro would not have written. A SEND MAPPING is silently DROPPED by both engines \u2014 `send mapping Mod.Export` parses, `mxcli check` passes, `exec` reports success, and nothing in the stored action references the mapping. Operation ARGUMENTS are dropped the same way", "cause": "sdk/mpr.serializeWebServiceCallAction was written without a Studio Pro reference and hardcodes five things it cannot know, and the codec engine's new writer reproduced it deliberately for parity. Measured against three Studio Pro-authored calls in ako/TestApp (Mendix 11.14.0, Clients.GetOrders / GetCustomerOrders / SaveOrder): ServiceName is the WSDL SERVICE name (\"OrdersWS\") not the local part of the imported service's qualified name (\"OrderSoapClient\"); ImportMappingCall.ContentType is \"Xml\" for a SOAP import mapping, not \"Json\"; Range.SingleObject follows cardinality (false for a list) rather than being always true; VariableType is the real result type (DataTypes$ObjectType with an Entity, DataTypes$BooleanType) rather than always DataTypes$VoidType; and a send mapping is Microflows$MappingRequestHandling {ContentType, MappingId, MappingVariableName}. Arguments live in RequestBodyHandling.ParameterMappings as Microflows$WebServiceOperationSimpleParameterMapping entries keyed by an escaped ParameterPath (\"http%3A//www.example.com/:GetOrder|OrderId\")", "file": "`sdk/mpr/writer_microflow_actions.go` (serializeWebServiceCallAction), `mdl/backend/modelsdk/microflow_webservice_write.go`", "insight": "**A guessed type name in a comment becomes a permanent refusal.** Legacy refused send mappings citing `Mendix$AdvancedRequestHandling`, said it 'requires a Studio Pro-generated example to determine the correct type storage name', and that refusal then shipped for as long as nobody went looking. The real type is `Microflows$MappingRequestHandling` \u2014 which THIS CODEBASE ALREADY WRITES for REST result/request handling \u2014 and the guessed name occurs in none of the three reference documents. The lesson is not about SOAP: when a writer refuses because a storage name is unknown, check whether a sibling feature already writes it before treating the refusal as a standing constraint. **Second, and the reason this was found at all: 'no reference exists' is a claim about where you looked.** The parity work asserted that no Studio Pro-authored SOAP document existed to pin against and used that to justify mirroring legacy; one existed in a separate repo the whole time (ako/TestApp, which carries both a consumed client and a published service). Byte-parity with what ships is a legitimate goal for a change scoped to stopping a silent drop \u2014 it is NOT evidence the shape is right, and conflating the two is how six defects got a passing test. Where a reference project exists, name it in the code so the next reader does not repeat the search", "refs": []} {"area": "mdl/backend", "date": "2026-09-11", "symptom": "`ALTER PAGE REPLACE`/`INSERT` inside a data view bound `datasource: selection ` re-scopes the new widget's attribute binding to the OUTER data view's entity (**CE1613** \"The selected attribute 'Mod.Outer.Attr' no longer exists\"), and inside a Gallery/DataGrid 2 sourced by a **microflow/nanoflow** drops it entirely (**CE0402** \"No value specified\", `describe` shows `ContentParams: [{1} = ]`). `mxcli check --references` and `exec` both report success; `CREATE PAGE` binds the same widget in the same position correctly", "cause": "The mutator resolved a widget's scope in TWO walks that each knew a different subset of the ten Forms$*Source kinds. `Forms$ListenTargetSource` carries no EntityRef at all \u2014 only the listen target's NAME \u2014 so the entity walk saw no source on the selection data view and left the context at the enclosing one. The flow walk (`findNearestDataSourceDoc`) read only a widget's TOP-LEVEL `DataSource` key, so a pluggable list \u2014 whose source sits at `Object.Properties[datasource].Value.DataSource` \u2014 contributed nothing, and its `Objects[].Properties[].Value.Widgets` descent (the one the entity walk gained in #935) was missing too", "file": "`mdl/backend/pagemutator/mutator.go` (`resolveSourceScope`/`resolveSourceScopeVia`, `listenTargetDataSource`, `widgetOwnDataSourceDoc`, `pluggableDataSourceDoc`; `EnclosingEntity`/`EnclosingEntityForChildren`/`EnclosingDataSourceFlow` now share the one walk `findNearestDataSourceDoc`, and `findEnclosingEntityContext` + its two helpers are deleted)", "insight": "**Count the source kinds before fixing one.** `generated/metamodel/types.go`'s `DataSource is implemented by` list closes the set at ten, and they divide exactly three ways \u2014 seven carry an EntityRef, two are flows, one (ListenTarget) borrows the scope of the widget it names \u2014 so one resolver can be complete, where three successive per-kind patches (FINDINGS #55 association+flow, #935 pluggable, this one) each left a hole. **A nearer source that resolves to no entity must SHADOW the outer one**: inheriting is what wrote the wrong entity, and it also mis-scoped a flow-sourced list nested in an entity-bound data view \u2014 a case the report did not name and the old code got wrong. The listen target is found by a shape-independent search for \"a document with this Name that has a data source\", which is what makes it work when the target is a pluggable widget keeping its source three levels inside its Object; a visited-set guards a hand-written listen cycle. **Measurement trap: a CE1613 SUPPRESSES the CE0402s in the same `mx check` run** \u2014 the first reading said mxbuild tolerated the unbound parameter, and the CE0402s only appeared once the re-scoped binding was fixed, so count bindings in `describe`, not errors. The issue's own second repro (a Gallery over a DATABASE source) no longer reproduced \u2014 #935 had fixed it \u2014 and the live defect was its flow-sourced variant, so re-measure a report against main before trusting its class. Tests `mdl/backend/pagemutator/mutator_selection_source_test.go` (7, incl. dangling/cyclic listen targets and the shadowing control); repro `mdl-examples/bug-tests/1076-alter-page-selection-and-flow-scope.mdl` \u2014 2 \u00d7 CE1613 + 3 unbound before, 0 errors after, on mxbuild 11.10.0. Each half proven load-bearing by stubbing it alone and rebuilding the CLI", "refs": ["mendixlabs/mxcli#1076", "#55", "#935"], "ce": ["CE0402", "CE1613"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 84c0659c8..a45c12459 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -578,7 +578,7 @@ {"area": "mdl/executor", "date": "2026-09-09", "symptom": "A microflow whose activity has a CUSTOM error handler comes back from `describe microflow` with the handler AND every activity in its branch missing. Output stays valid MDL, `mxcli check` passes, nothing on stderr \u2014 so describe\u2192edit\u2192exec deletes the handler from the model. Reported on a create-variable activity with \"custom with rollback\" (v0.21, Mx 11.12.3); reproduced on HEAD/11.14.0 with a pure mxcli round trip", "cause": "TWO STACKED DEFECTS. (1) `getActionErrorHandlingType` was a hand-maintained switch covering 17 of the 38 action types that store ErrorHandlingType. `emitActivityStatement` walks the error branch only when `hasCustomErrorHandler(errType)` agrees, so each of the 21 missing types lost the ENTIRE `on error { \u2026 }` block, not just the suffix \u2014 CreateObject and ChangeObject among them. (2) legacy only: 9 parse functions never read ErrorHandlingType off the BSON, so the value was gone before the describer was asked. Fixing (1) alone left legacy still broken, which is how they hid each other", "file": "`mdl/executor/cmd_microflows_show_helpers.go` (`getActionErrorHandlingType` \u2192 new `actionErrorHandlingField`); `sdk/mpr/parser_microflow.go`, `sdk/mpr/parser_microflow_actions.go` (9 parsers); grammar+visitor+builder for 8 statements; `mdl/executor/validate_microflow_error_handling.go` (MDL076 table, new MDL077)", "insight": "**Count the gap before fixing the instance** \u2014 the reported activity was 1 of 21, measured by diffing the switch's cases against the action types declaring the field. **Replace the enumeration, do not extend it**: the list had already been patched per-instance (#863) and silently regrew, so it is now a reflection lookup on the `ErrorHandlingType` field, with `RestOperationCallAction` the ONE deliberate exclusion (Mendix rejects a custom handler there, CE6035). Actions embed model.BaseElement, which has no such field, so no promoted field is picked up by accident. **The trap: fixing the describer alone makes things WORSE.** 8 statement forms had no `onErrorClause` in the grammar \u2014 `declare` (the reporter's own), `set`, `change`, `log`, `show page`, `close page`, `show message`, `validation feedback` \u2014 so DESCRIBE began emitting `declare $name String = 'v' on error { \u2026 };`, which fails to parse (`mismatched input 'on' expecting ';'`). Trading a silent drop for a broken script is not a fix; the grammar was extended instead. **Measured on 11.14.0, and the result is not guessable**: all 8 accept a custom handler (0 errors), but for `on error continue` create-VARIABLE and change-VARIABLE are fine while change-OBJECT, log, show page, close page, show message and validation feedback are CE6035 \u2014 now in MDL076's deny-list, whose comment claiming Log/Change were \"unreachable from a script\" this change invalidated. New MDL077 refuses `on error` on the list-operation/aggregate forms of `set`, which genuinely have no ErrorHandlingType in the metamodel \u2014 one MDL keyword spanning activities that can and cannot hold the clause. **A non-terminating handler causing CE0108 is NOT a bug**: the branch merges back and a later variable is out of scope on the error path; Studio Pro reports the same. Controls: revert the lookup \u2192 all 7 describer cases fail naming the dropped branch; revert one parser \u2192 the legacy case fails; a no-clause microflow must still render NO suffix (#840 in reverse). Repro `mdl-examples/bug-tests/microflow-1078-error-handler-roundtrip.mdl`; describe\u2192exec\u2192describe byte-identical and reported \"Unchanged microflow\"", "refs": ["#1078", "#863", "#840"], "ce": ["CE6035", "CE0108"]} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "CI `make test-integration` fails where `go test ./...` is green: `TestMxCheck_DoctypeScripts/02b-nanoflow-examples.mdl` (both engines) and `/03-page-examples.mdl` report **CE6035 \"Error handling type is not supported\"** on 11 activities \u2014 every un-annotated Change object / Log message / Validation feedback / Close page in a NANOFLOW. Self-inflicted while fixing #1078", "cause": "Eight builders were switched from `fb.ehType(nil)` to `explicitErrorHandling(fb, s.ErrorHandling)` on the action. `explicitErrorHandling` returns EMPTY for \"no clause\", and the writers turn empty into a literal `\"Rollback\"` via orDefault \u2014 but `fb.ehType(nil)` is CONTEXT-DEPENDENT and returns **Abort** in a nanoflow. So every un-annotated nanoflow activity silently changed from Abort to Rollback, which mxbuild rejects", "file": "`mdl/executor/cmd_microflows_builder_actions.go` + `cmd_microflows_builder_calls.go` (8 sites reverted to `fb.ehType(s.ErrorHandling)`); `mdl/executor/nanoflow_validation.go` (`getErrorHandling`)", "insight": "**The same helper is correct for one action and wrong for its neighbour, and the difference is what the code did BEFORE.** `explicitErrorHandling` is right for Retrieve/Delete (#1020-era): their writers emitted a hardcoded `\"Rollback\"` that those two actions accept in every flow flavour, so empty\u2192Rollback is a no-op. It is wrong wherever the builder already supplied a *context-dependent* default \u2014 empty discards the flow flavour. The rule: before replacing a default-supplying expression, ask what the OLD expression returned in every context, not just the one you are testing. Its own doc comment names Abort/nanoflow and I still missed it, because I was reading it as \"the safe choice\" rather than \"the choice that preserves THIS call site's prior value\". **`go test ./...` does not run this suite** \u2014 the mx-check round trips are behind `-tags integration` (`make test-integration`, ~30 min), so a green unit suite says nothing about whether mxbuild still accepts what mxcli writes; run it before pushing anything that touches a serialized default. Regression test `TestAuthorOnError_NanoflowKeepsAbortWithoutAClause` (control: the same statements in a microflow must NOT become Abort) \u2014 a unit test, so it catches this in seconds instead of 30 minutes. A second, quieter gap in the same change: `getErrorHandling` in nanoflow_validation.go gates the walk that looks for disallowed actions INSIDE a handler body, so the eight new statements had to be added there too or a Java action nested in `declare \u2026 on error { \u2026 }` goes unreported (test carries a `commit` control, since that entry point is `validateNanoflowBody`, NOT the exported `ValidateNanoflowBody`, which is a different check)", "refs": ["#1078"], "ce": ["CE6035"]} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "In a NANOFLOW, `log ... on error continue|rollback|{ \u2026 }`, and a custom handler on change/show page/close page/show message/validation feedback, are written by mxcli and rejected by mxbuild as **CE6035 \"Error handling type is not supported\"**. `mxcli check` passed. Reachable for the first time via mendixlabs/mxcli#1078, which gave those statements an onErrorClause", "cause": "No rule covered it. MDL076 runs on the microflow validator, which has no flow flavour, so it cannot express \"CE6035 in a nanoflow but fine in a microflow\" \u2014 and every one of these six IS fine in a microflow with a custom handler", "file": "`mdl/executor/nanoflow_validation.go` (`checkNanoflowErrorHandling`, `nanoflowErrorHandlingUnsupported`, wired into `validateNanoflowStatements`)", "insight": "**A nanoflow accepts error handling on almost nothing**: measured on 11.14.0, only the two VARIABLE activities (create-variable, change-variable) take a clause; the other six are CE6035 whichever form is written, because a nanoflow activity's only accepted value is Abort \u2014 the no-clause default, which no MDL syntax writes. So the rule refuses the CLAUSE, not one spelling. The split is by ACTIVITY, not by client-side/server-side: `show message` is as client-side as it gets and still refuses one, while `declare` accepts it. Same permissive pair as `continue` in a microflow, which is the only pattern visible across both tables. **Two validators, two entry points, easy to wire to the wrong one**: `validateNanoflowBody` (unexported) is the disallowed-action walk that exec runs; `ValidateNanoflowBody` (exported) is the variable/semantic check that `check` runs. A test targeting the exported one passes vacuously \u2014 caught only because the test carried a `commit` control that also failed. Note the pre-existing consequence: every nanoflow restriction, including the 22 disallowed action types, is enforced at EXEC and not by `mxcli check` (even with `--references`) \u2014 a check/exec divergence worth closing on its own, but not in a bug fix, since it would newly reject scripts that pass check today", "refs": ["#1078"], "ce": ["CE6035"]} -{"area": "mdl-executor", "date": "2026-09-10", "symptom": "`mxcli diff-local` prints `Summary: 0 new, 1 modified, 0 deleted` and not a single line of diff \u2014 in all three formats \u2014 for an edited microflow on the default engine", "cause": "`Backend.ParseMicroflowFromRaw` was also unimplemented on the modelsdk engine and returned nil. `microflowBsonToMDL` guards nil by substituting a `-- parse failed --` stub, but the stub text is a CONSTANT, so both sides of the diff rendered the same string and the differ found zero hunks.", "file": "mdl/executor/cmd_diff_local.go", "insight": "A nil-guard written to degrade visibly degrades INVISIBLY when both sides of a comparison degrade identically \u2014 the fallback has to vary with its input, or it erases the difference it was meant to surface. The same shape bit the attribute renderer in the same command for an unrelated reason: attributeBsonToMDL read the type object from raw[\"Type\"], but Mendix stores it under the STORAGE name \"NewType\" (the per-property half of the storage-name split in CLAUDE.md), so every attribute rendered \"Unknown\" on BOTH engines and narrowing String(200) to String(50) diffed to nothing. Measurement that settled it: dump the unit's decoded top-level keys instead of trusting the field name \u2014 `Attributes` is a primitive.A whose FIRST element is an int32 typed-array marker, and each attribute map has NewType, no Type at all.", "refs": ["mendixlabs/mxcli#1080"]} +{"area": "mdl/executor", "date": "2026-09-10", "symptom": "`mxcli diff-local` prints `Summary: 0 new, 1 modified, 0 deleted` and not a single line of diff \u2014 in all three formats \u2014 for an edited microflow on the default engine", "cause": "`Backend.ParseMicroflowFromRaw` was also unimplemented on the modelsdk engine and returned nil. `microflowBsonToMDL` guards nil by substituting a `-- parse failed --` stub, but the stub text is a CONSTANT, so both sides of the diff rendered the same string and the differ found zero hunks.", "file": "mdl/executor/cmd_diff_local.go", "insight": "A nil-guard written to degrade visibly degrades INVISIBLY when both sides of a comparison degrade identically \u2014 the fallback has to vary with its input, or it erases the difference it was meant to surface. The same shape bit the attribute renderer in the same command for an unrelated reason: attributeBsonToMDL read the type object from raw[\"Type\"], but Mendix stores it under the STORAGE name \"NewType\" (the per-property half of the storage-name split in CLAUDE.md), so every attribute rendered \"Unknown\" on BOTH engines and narrowing String(200) to String(50) diffed to nothing. Measurement that settled it: dump the unit's decoded top-level keys instead of trusting the field name \u2014 `Attributes` is a primitive.A whose FIRST element is an int32 typed-array marker, and each attribute map has NewType, no Type at all.", "refs": ["mendixlabs/mxcli#1080"]} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A CONTAINER inside a DataGrid 2 control bar with `Action: nanoflow M.NF`, where the nanoflow declares a required entity parameter, built to **CE1571** \"No argument has been selected for parameter 'X'\" at Container 'containerUnlinkMat' — while `mxcli check --references` printed \"Check passed!\". Reported as mendixlabs/mxcli#1082, whose stated root cause was that MDL had no syntax for passing an argument through a container action", "cause": "The CE1571 argument check (`validateDataSourceArguments`, now `validateFlowArguments`) walked `w.GetDataSource()` only, so the identical missing-argument fault on an ACTION slot was silent. Separately the context walk let a control bar inherit its data widget's row context, so even once actions were walked the reported shape would still have passed", "file": "`mdl/executor/validate_datasource_args.go` (`argDiff`, `widgetActions`, `actionArgErrors`, `isControlBar` + the per-child walk); `mdl/executor/validate_page_button_context.go` (MDL-BUTTON01 suggestion); `cmd/mxcli/syntax/features_page.go` (page.action topic)", "insight": "**The report's root cause was wrong and the real defect was one layer up — check the premise by executing it, not by reading it.** `actionExprV3` has carried `NANOFLOW qualifiedName microflowArgsV3?` all along and `buildContainerV3` hands the action to the same `buildClientActionV3` a button uses; three pages in one script settled it (bare action → CE1571; `($P = $dgMaterials)` on the container → 0 errors; same on an actionbutton → 0 errors), and dropping the bare page took the project 1 → 0 errors as the control. What was broken is that nothing SAID so, and the author concluded from `mxcli syntax page.action` — which listed `Action: NANOFLOW Module.NF` with no argument variant two lines under `Action: MICROFLOW Module.MF(Param: $val)` — that the syntax did not exist. A silent check plus an incomplete syntax topic reads as a missing feature. **A control bar is not row-scoped, and that belongs in the WALK, not in either rule**: six containers carrying one identical fault, one `mx check` on 11.12.0, gave column → no error / control bar → CE1571, and the same pair holds for a DATA SOURCE (dataview in the control bar → CE1571, in a column → clean), so fixing it in the shared walk closed a latent false negative on the older half of the rule too. It drops only the data widget's OWN object — measured, an outer dataview's context still reaches the control bar — so the control bar is walked with its parent's incoming context, not an empty one. **Enumerate action slots by sweeping Properties for `*ast.ActionV3`, never by a key list**: `Action`/`OnClick` share one key, `OnChange` has its own, and a pluggable widget's named slot uses its own name; a key list would have fixed the reported slot and reproduced the bug on the rest. **The repro cannot be a .fail.mdl**: the rule needs a project to read the flow's signature, and `make check-mdl` runs check with no project, so a negative fixture would report a working rule as regressed (the #891/#892 trap)", "issue": "mendixlabs/mxcli#1082"} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A microflow note (Microflows$Annotation) connected to several activities came back DUPLICATED after describe -> exec: one note with N AnnotationFlows became N notes with one flow each. Measured on Mendix 11.13.0: 1 note + 2 flows -> 2 notes. mx check: 0 errors on both sides, so nothing warned.", "cause": "buildAnnotationsByTarget joined AnnotationFlows to Annotation objects and kept only the CAPTION, filing it under each flow's destination. The note's identity was discarded at that join, so the describer had no way to know two lines were the same note, and MDL had no way to say so either: ActivityAnnotations carried AnnotationText as a single STRING per activity. attachAnnotation then minted a fresh Annotation + flow per line.", "file": "mdl/executor/cmd_microflows_show_helpers.go (buildAnnotationsByTarget, annotationEmitter), mdl/executor/cmd_microflows_builder_annotations.go (attachAnnotation, defaultAnnotationGeometry), mdl/ast/ast_microflow.go (MicroflowAnnotation), mdl/visitor/visitor_microflow_statements.go (parseNoteAnnotation), mdl/grammar/domains/MDLSettings.g4 (annotationParamName)", "insight": "The relation is MANY-TO-MANY and a per-activity string lost BOTH directions — which is how a duplication report turned out to be hiding a deletion. One note to N activities duplicated (the report); N notes to one activity kept only the LAST, because the visitor assigned into the single slot. The second is strictly worse and nobody had noticed it. When a report says a round trip 'duplicates' something, check the transpose before fixing: the same missing cardinality usually destroys in the other direction. The fix that holds is giving the construct IDENTITY in the language (`id:`) rather than deduplicating on content — merging two notes because their text matches would be a second silent rewrite, so the control test is two identical-text notes that must STAY two.", "refs": ["mendixlabs/mxcli#1077"], "ce": []} {"area": "mdl/executor", "date": "2026-09-10", "symptom": "A parameterised annotation whose key is a lexer keyword parses and is SILENTLY IGNORED. `@annotation(text: 'x')` and `@annotation(position: (1, 2))` were accepted by the grammar, reached no visitor case, and vanished; `@annotation(size: (1, 2))` and `@annotation(zz: (1, 2))` worked.", "cause": "annotationParam is `annotationParamName COLON value | annotationValue`, and annotationParamName lists only IDENTIFIER plus a hand-maintained set of keywords (FROM, TO, TRUE, FALSE, TAIL). A keyword key does not FAIL the parse — it falls through to the positional alternative — so the parameter is consumed and means nothing.", "file": "mdl/grammar/domains/MDLSettings.g4 (annotationParamName), mdl/visitor/visitor_microflow_statements.go", "insight": "The failure mode of that grammar rule is silence, not a parse error, so a new annotation parameter cannot be assumed to work because a probe script parsed clean — probe that the VALUE arrives, not that the text is accepted. Words that are already MDL keywords are exactly the readable ones you reach for (`text`, `position`, `size`, `caption`), so this will keep recurring; anything added must be listed in annotationParamName. The visitor now records an unusable parameter on ActivityAnnotations.InvalidNotes and MDL079 refuses it, matching how @curve's InvalidCurves feeds MDL060.", "refs": ["mendixlabs/mxcli#1077", "mendixlabs/mxcli#884"], "ce": []} diff --git a/docs-wiki/SYNC_LOG.md b/docs-wiki/SYNC_LOG.md index 74275c034..57a645d1d 100644 --- a/docs-wiki/SYNC_LOG.md +++ b/docs-wiki/SYNC_LOG.md @@ -55,3 +55,6 @@ not capture, because sources are upstream of the commit. | 2026-08-31 | bug-patterns/expression-translation-drift.md | .claude/skills/fix-issue/findings/mdl-visitor.jsonl (13 records), mdl/visitor/visitor_microflow_expression.go, mdl/visitor/visitor_helpers.go | New page (added to seed table). First pass over mdl/visitor. Distinct from platform-semantics-gaps: there the MDL is illegal Mendix, here the MDL is correct and the TRANSLATION says something else — the worst case a microflow computing a different number with every check green. Records the ANTLR hidden-token trap and why the same expression text means different things in different slots | | 2026-08-31 | bug-patterns/misleading-diagnostics.md | .claude/skills/fix-issue/findings/mdl-visitor.jsonl (6 records), mdl/visitor/visitor.go | New page (added to seed table). Graded above "the message could be clearer": a wrong hint costs however long the reader spends acting on it, and in the reported cases they blamed their quoting, renamed an attribute that was fine, or concluded a construct was unsupported. A hint's precision matters more than its coverage | | 2026-08-31 | bug-patterns/visitor-wiring-gaps.md | .claude/skills/fix-issue/findings/mdl-visitor.jsonl, mdl/visitor/visitor_enumeration.go, mdl/visitor/visitor_helpers.go | **Re-sync** (first since the 2026-05-24 initial synthesis). Broadened from one size of gap to three — a field, a structure (ELSIF arms lowered into nested ifs), and a whole statement that parses and dispatches to nothing — and added the neighbouring failure where a field is wired to the WRONG thing, which reports success and changes meaning. sources: updated from the findings directory to the specific shard; wiki-links normalised to bare slugs | +| 2026-09-12 | bug-patterns/check-mxbuild-drift.md | .claude/skills/fix-issue/findings/mdl-executor.jsonl + mdl-backend.jsonl + mdl-grammar.jsonl (55 records dated after the last sync), mdl/executor/validate_program.go, docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md | **Re-sync.** 90 mdl/executor findings had landed since 2026-08-31 and this was the dominant class. Three new sections. (1) *Three answers, not two*: a check needs resolved / not-resolved / **could not establish**, where exec needs two — turning "I could not look" into "your attribute is missing" is a false error that blocks a script which builds cleanly, and the guards are non-obvious (a module the script creates; an EMPTY listing meaning the backend could not answer). Includes that `check` normally runs with **no project**, which is also the CI default, and the allow-list/deny-list choice for type-keyed rules. (2) *Don't model what you can run*: the mutator-probe technique — run the real setter against a throwaway deep copy whose Save is refused — as the strongest form of the existing "both passes or neither". (3) *The instruments lie in specific ways*: one error per microflow hides the one being measured; a load failure suppresses the error-count line while still exiting non-zero (third occurrence); the corpus sweep has a noise floor (11 of 515 scripts nondeterministic) and is blind to a misparse, since it diffs diagnostics. Also added a remedy beat — two rules shipped whose `Fix:` line broke a working project | +| 2026-09-12 | bug-patterns/widget-type-object-drift.md | .claude/skills/fix-issue/findings/ mdl-executor.jsonl + cmd-mxcli.jsonl + sdk.jsonl + mdl-backend.jsonl (49 CE0463/CE3637 records across the corpus), .claude/skills/diagnose-ce0463.md, .claude/skills/debug-bson.md, sdk/widgets/templates/README.md, sdk/mpr/writer_widgets.go | **Re-sync**, first since 4e185f73 and the oldest page in the category. `sources:` **added** diagnose-ce0463.md (written after the last sync; debug-bson.md's own header defers to it) and replaced the bare findings *directory* with the four shards that actually carry the records. The page had none of the framing that makes the class tractable: that **two unrelated bugs wear this error** (package upgraded after authoring — not an mxcli defect — versus authored-fresh-and-wrong) and that the two controls separating them are not optional; that `mxcli docker check` runs `mx update-widgets` first and so reports 0 errors on a project that has a CE0463; that a difference from the reference is not a cause (25 differences, 3 patched in isolation, none moved the count). Added the two new findings' contribution: evaluate a visibility rule against the configuration that will be WRITTEN rather than the one the package declares, and compare enum values against the .mpk's `enumerationValue` keys, not captions | +| 2026-09-12 | (no page) findings `area` normalisation | .claude/skills/fix-issue/findings/mdl-executor.jsonl, mdl-backend.jsonl | Not a sync. Two records written 2026-09-10 used the SHARD FILENAME (`mdl-executor`, `mdl-backend`) as their `area` where all 677 others use the path (`mdl/executor`, `mdl/backend`). `make check-findings` passes either way — it requires the field, not a vocabulary — so `make digest-status` had silently bucketed both into its "(32 areas < 5)" row. Corrected in place before using those counts to choose this run's scope | diff --git a/docs-wiki/bug-patterns/check-mxbuild-drift.md b/docs-wiki/bug-patterns/check-mxbuild-drift.md index bb7a784d7..219da9ce9 100644 --- a/docs-wiki/bug-patterns/check-mxbuild-drift.md +++ b/docs-wiki/bug-patterns/check-mxbuild-drift.md @@ -1,9 +1,11 @@ --- title: When `mxcli check` and mxbuild Disagree category: bug-pattern -last-synced: ced830e0 +last-synced: 392cacd6 sources: - .claude/skills/fix-issue/findings/mdl-executor.jsonl + - .claude/skills/fix-issue/findings/mdl-backend.jsonl + - .claude/skills/fix-issue/findings/mdl-grammar.jsonl - mdl/executor/validate_program.go - docs/11-proposals/PROPOSAL_check_mxbuild_gap_heuristics.md --- @@ -29,6 +31,8 @@ reports an **error**, a false positive is a blocker rather than a warning. ## How it fits +### Getting the prediction right + **Verify a rule against mxbuild, not against intuition.** Rules have been added on a plausible reading of a CE code and later measured to be wrong: one flagged format functions over association navigation as a build error and was deleted @@ -40,18 +44,8 @@ be checked against mxbuild. deleted rule was written after reproducing several failures that shared the construct it flagged — and the construct was not the cause. They shared a *different* hidden defect, in the write path. When a write-path fix lands, -re-validate the checks that were derived from the same symptoms; a -correlation-based rule outlives the correlation. - -**Mirror the builder's own condition rather than inventing a second one.** Where -`check` predicts something the builder decides, the two conditions must be the -same expression, not two readings of the same intent — otherwise they drift on -the first edit to either. - -**Run a new rule over `mdl-examples/` before wiring it up.** One candidate hit 4 -of 374 example files and 3 of the hits were false positives, because the rule -read the AST while the outcome depended on what the builder synthesises. The -corpus is the cheapest false-positive test available. +re-validate the checks derived from the same symptoms; a correlation-based rule +outlives the correlation. **Severity turns on what the builder's condition is actually about, and the intuitive reading is often the wrong one.** The empty-outcome rule looked like a @@ -62,15 +56,67 @@ which makes it an error. Measure the exemption you are about to grant; do not infer it. **Severity is the design decision, not an afterthought.** A rule whose vocabulary -cannot be proven complete — anything about widget properties, or about a name -that might be legal in a context the rule cannot see — must be a *warning*, or -it trades a silent defect for a false refusal. That only works if warnings -genuinely do not block: an exec guard written as `if len(violations) > 0` makes -every warning fatal, which is how a warning-severity rule silently became a -blocker for everything it touched. - -**Close a gap in both passes or in neither.** A check-time rule does not protect -a script that runs `exec` directly, and `--no-check` exists. Both call the same +cannot be proven complete — anything about widget properties, or a name that +might be legal in a context the rule cannot see — must be a *warning*, or it +trades a silent defect for a false refusal. That only works if warnings genuinely +do not block: an exec guard written as `if len(violations) > 0` makes every +warning fatal, which is how a warning-severity rule silently became a blocker for +everything it touched. + +**The remedy is part of the rule, and a wrong one is worse than silence.** Two +rules have shipped whose `Fix:` line walked a working project into a broken one — +a view-entity column told to change `Integer` to `Decimal` (the reverse of what +mxbuild wants), and a design-property warning whose suggested keys turned 16 +warnings into 17 `CE6083` errors. A rule that cries wolf costs attention; a rule +that hands over a wrong remedy costs the build. Where a fallback has to guess, +return *unknown*: a skipped column is a missed error, a wrong guess is a +manufactured one. + +### Three answers, not two + +**A check needs three outcomes where `exec` needs two.** `exec` can answer +resolved / not-resolved, because it has a fallback either way. A check that turns +*I could not look* into *your attribute is missing* is a false error blocking a +script that builds cleanly — so **could not establish** is a third state, and only +the middle one is reported. The guards that implement it are never obvious from +the rule: a module the script itself creates has no listing to resolve against +yet; an **empty** listing means the backend could not answer, not that the project +has none; a qualified member path can only be judged once the base entity is +known. Each of those was a measured false positive, not a hypothetical. + +**The third state has to be applied at every reporting site, not once.** One +member-reference rule inherited the discipline on its bare-name path and not on +its qualified path, so exactly half of it was safe. + +**`check` normally runs with no project at all, and that is the CI default.** +`make check-mdl` sweeps the corpus without `-p`, where the widget registry holds +only the embedded definitions and every real project's widget looks unknown. A +rule measured only with a project is a rule measured in the minority case: one +went from clean to 14 violations on a single example and broke seven corpus files. + +**For a rule keyed on a type, prefer a deny-list.** An allow-list makes the +unknown case an *error*; a deny-list makes it silence. A missed warning costs +nothing and a false one tells an author their working page is broken. + +### Don't model what you can run + +**Mirror the builder's own condition rather than inventing a second one.** Where +`check` predicts something the builder decides, the two conditions must be the +same expression, not two readings of the same intent — otherwise they drift on +the first edit to either. This is the [[duplicate-resolver-drift]] class pointed +at mxbuild. + +**Better still, run the real operation against a throwaway copy.** The vocabulary +of an `ALTER … SET` is partly a switch in the mutator and partly the *stored* +widget's own PropertyTypes, which belong to whatever package the project +installed — no registry in this repo can state it for an arbitrary project. So +the check opens the document and runs the actual setter against a deep copy whose +`Save` is refused, keeping only the error. Check and exec cannot drift, because +there is one resolver, and the author gets exec's exact wording from the +pre-flight. + +**Close a gap in both passes or in neither.** A check-time rule does not protect a +script that runs `exec` directly, and `--no-check` exists. Both call the same function so they cannot diverge — the convention exists because they did. **When probing a gap, probe every sibling.** A missing reference check is almost @@ -79,12 +125,40 @@ validated — called microflow, called workflow, user task page, targeting microflow, context entity, and the workflow's own module. Fixing the reported one leaves the class open and the next report looks new. -**Grade by consequence.** The same statement can produce a recoverable build -error or an unopenable project depending on one detail — a qualified-but-missing -name versus an unqualified one. Those want different gates: the first needs a -project and belongs in the reference check; the second is a static property of -the statement and can be refused with no project at all, which is also what makes -it testable as a `.fail.mdl` fixture in CI. +**Grade by consequence.** The same statement can produce a recoverable build error +or an unopenable project depending on one detail — a qualified-but-missing name +versus an unqualified one. Those want different gates: the first needs a project +and belongs in the reference check; the second is a static property of the +statement and can be refused with no project at all, which is also what makes it +testable as a `.fail.mdl` fixture in CI. The severe end of that scale is +[[unloadable-model-writes]]. + +### The instruments lie in specific ways + +Every rule here is justified by a measurement, so the measurement apparatus is +part of the class. + +**mxbuild reports one error per microflow, so a second defect in the same document +hides the one you are measuring.** An exemption justified by *measured: builds +clean* is only sound if the reproduction was otherwise valid. When a measurement +says a construct is clean, add the minimum that removes every *other* error from +that document and measure again; the differential is what settles it. + +**A load failure suppresses the error-count line while still exiting non-zero.** +`mx check` prints a stack trace and no `The app contains: N errors.` — so a +harness grepping for that line reads the run as inconclusive, and a human reads it +as success. Never conclude success from the absence of that line; read the exit +code. This has now cost a diagnosis three times. + +**The corpus sweep has a noise floor and a blind spot.** Running a new rule over +`mdl-examples/` before wiring it up is still the cheapest false-positive test +available — one candidate hit 4 files and 3 hits were false positives. But the +output was *nondeterministic* until the validators that emit per map key sorted +them, and 11 of 515 scripts differed between runs of the same binary; and a diff +of `check` output compares **diagnostics**, so it is blind to a construct that +parses into the wrong AST shape rather than being rejected. There, 515 scripts +said nothing and two visitor unit tests caught it immediately. Diff diagnostics to +find false positives; assert on the AST to find misparses. ## See also @@ -92,5 +166,8 @@ it testable as a `.fail.mdl` fixture in CI. predicate, its CE code, and the measurement behind it - [[unloadable-model-writes]] — the failures that have no CE code because the build never gets that far +- [[duplicate-resolver-drift]] — the general shape when the second answer is not + mxbuild's +- [[misleading-diagnostics]] — what a wrong message costs once it is believed - `PROPOSAL_check_mxbuild_gap_heuristics.md` — the design rationale for predicting mxbuild at all diff --git a/docs-wiki/bug-patterns/widget-type-object-drift.md b/docs-wiki/bug-patterns/widget-type-object-drift.md index 28332c3e0..94b54b2c5 100644 --- a/docs-wiki/bug-patterns/widget-type-object-drift.md +++ b/docs-wiki/bug-patterns/widget-type-object-drift.md @@ -1,30 +1,108 @@ --- title: Widget Type / Object Drift (CE0463) category: bug-pattern -last-synced: 4e185f73 +last-synced: 392cacd6 sources: - - .claude/skills/fix-issue/findings/ + - .claude/skills/fix-issue/findings/mdl-executor.jsonl + - .claude/skills/fix-issue/findings/cmd-mxcli.jsonl + - .claude/skills/fix-issue/findings/sdk.jsonl + - .claude/skills/fix-issue/findings/mdl-backend.jsonl + - .claude/skills/diagnose-ce0463.md - .claude/skills/debug-bson.md - sdk/widgets/templates/README.md - sdk/mpr/writer_widgets.go --- -> **Do not duplicate**: the specific CE0463 fix recipes live in the `fix-issue/findings/*.jsonl` records, the diff workflow lives in `.claude/skills/debug-bson.md`, and the template-extraction procedure lives in `sdk/widgets/templates/README.md`. This page describes the pattern only. +> **Do not duplicate**: the elimination order, the two controls and the +> measurement traps are canonical in `.claude/skills/diagnose-ce0463.md`; the +> general BSON diff workflow is in `debug-bson.md`; template extraction is in +> `sdk/widgets/templates/README.md`; the per-instance fixes are in the findings. +> This page describes why the class behaves the way it does. ## What this is -A family of pluggable-widget bugs (DataGrid2, ComboBox, Gallery, filter widgets) where the embedded `WidgetType` and `WidgetObject` drift out of structural sync. Studio Pro detects the mismatch and raises CE0463 "the definition of this widget has changed" — and on master-detail pages that cascades into CE3637 on the dependent DataView. +A pluggable widget stores two coupled structures: the **type** (the PropertyTypes +schema — what properties exist) and the **object** (the WidgetObject — the values). +Studio Pro requires them to correspond exactly: every PropertyType needs its +WidgetProperty, every `TypePointer` must resolve, ordering must match, and no +field may appear that the reflection schema does not declare. Any deviation is +CE0463 *"the definition of this widget has changed"* — which on a master-detail +page cascades into CE3637 on the dependent DataView. + +Nearly fifty findings carry this code, which makes it the most-reported single +error in the repo. It is expensive out of proportion to its frequency, and the +reason is in the wording: **CE0463 names the widget *version*, and is caused by +almost anything in the widget's stored BSON.** The error points at the one thing +that is usually fine. ## How it fits -A pluggable widget stores two coupled structures: the `type` (the PropertyTypes schema — what properties exist) and the `object` (the WidgetObject — the actual values). Studio Pro enforces that they match exactly: every PropertyType needs a corresponding property, cross-reference `TypePointer`s must resolve, ordering must match, and no field may appear that is absent from the reflection schema. Any deviation marks the definition as "drifted." +**Two unrelated bugs wear this error, and separating them is the first move.** +Either the widget *package* changed after the widgets were authored — a stored +instance now carries a property the new definition dropped, which is what Studio +Pro's "Update all widgets" exists for and is **not an mxcli defect** — or the +widgets were authored against the current package and mxcli emitted something it +does not accept. The two are indistinguishable in `mx check` output. Two controls +tell them apart: whether Studio Pro's *own* template widgets fail alongside yours, +and whether `mx update-widgets` clears it. Skipping that step produces a confident +wrong answer, and has. + +**The default measurement hides the class.** `mxcli docker check` runs +`mx update-widgets` first, on purpose, to suppress exactly the package-drift noise +above — so it reports 0 errors on a project that genuinely has a CE0463, *and* +repairs the project on the way past, leaving it clean for any later check too. +Both of this cycle's real CE0463s were invisible until someone passed +`--no-update-widgets`. (`mxcli check` is not an alternative: it validates an MDL +script and never invokes mxbuild, so it cannot produce CE0463 at all.) + +**The cause is usually a value, not the schema** — four of five fixes in one +release cycle were value-shaped while the error named the version: an empty +`TextTemplate` where Studio Pro stores the attribute name, an empty +`Forms$ClientTemplate` where Studio Pro stores `null`, a placeholder `" "` for an +unset string. Property-set drift against the package is real but a weak predictor: +one widget needs nineteen additions and passes, while another is byte-for-byte in +sync and fails. + +**A difference from the reference is not a cause.** With one grid's type +byte-identical to `mx update-widgets` output, twenty-five value-level differences +remained; three were patched in isolation and none moved the error count. The diff +*bounds* the search, it does not rank it — budget for that rather than +pattern-matching the first plausible entry. + +**A visibility rule must be evaluated against the configuration that will be +written, never the one the package declares.** The two diverge precisely on +unmapped properties, which is the only place the rule matters — which is how every +mxcli-authored Image widget failed a build while the package's own defaults said +it should not. The corollary for fixtures: a test whose helper restates the +default under test is not a fixture, it is the hypothesis wearing a test's +clothes. + +**Where a property is an enumeration, compare against the package's +`enumerationValue` *keys*, not its captions.** Both sides are plain strings, so +nothing else compares them and a caption-shaped value is stored and built without +complaint until Studio Pro reads it. A wrong value hidden inside the default +configuration will also not reproduce at all — vary the property that unhides it, +or the measurement is of nothing. -The drift recurs because the two structures are written from different code paths, so it is easy for one to gain or lose a field the other does not. Concrete triggers include emitting a field outside the reflection schema (the `TimeFormat` regression in `Forms$FormattingInfo`, see [`sdk/mpr/writer_widgets.go`](../../sdk/mpr/writer_widgets.go)), property ordering mismatches, and `null` where a `Forms$ClientTemplate` is required. The tell-tale is CE0463 on a widget you just wrote via MDL. +**The drift recurs because the two structures are written from different code +paths**, and the definitions themselves exist in more than one copy (the embedded +set, the modelsdk set, and the project's own `.mxcli/widgets/`). A fix applied to +one copy is latent in the others — the same [[duplicate-resolver-drift]] shape, +here in data rather than code. -A key trap: `mx check` is tolerant and passes anyway — only `mx diff` and Studio Pro are strict — so green checks do not mean the project opens. The diagnostic is the "Update widget" diff: let Studio Pro re-save the widget and diff its output against yours. The per-trigger recipes are in the symptom table; the diff methodology is in [`debug-bson.md`](../../.claude/skills/debug-bson.md). +**`mx check` is the wrong gate for this class.** It is tolerant of both extra and +missing properties; `mx diff` and Studio Pro are strict, and Studio Pro runs +`mx diff` internally when opening a project with uncommitted changes. A green +check does not mean the project opens. ## See also -- [fix-issue findings](../../.claude/skills/fix-issue/findings/) — the per-instance CE0463/CE3637 fix recipes -- [[architecture/widget-engine]] — how `.def.json` definitions and templates produce widget BSON -- [[models/version-gating]] — why a field can be valid in one Mendix minor and drift in another +- [`diagnose-ce0463.md`](../../.claude/skills/diagnose-ce0463.md) — the + elimination order, the controls, and what each normalisation hides +- [fix-issue findings](../../.claude/skills/fix-issue/findings/) — the + per-instance CE0463 / CE3637 fixes +- [[architecture/widget-engine]] — how `.def.json` definitions and templates + produce widget BSON +- [[duplicate-resolver-drift]] — why a definition kept in three places drifts +- [[models/version-gating]] — why a field can be valid in one Mendix minor and + drift in another From 48f4cb251100ec1110338417431e8a3e1e526187 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 20:21:49 +0000 Subject: [PATCH 13/18] fix(microflow): give an end event two paths reach a merge to join them at MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `describe microflow` → `exec` round trip of all 41 microflows in ako/TestApp produced a project mxbuild refuses: CE0709 "Sequence flow is not accepted by origin or destination", naming neither the microflow nor the edge. `mxcli check` passed, `exec` reported success, and the project still opened in Studio Pro — the document is well formed, just over-connected — so only a build caught it. It was filed as unexplained "flow-graph drift"; this is the diagnosis. A Mendix end event accepts exactly ONE incoming sequence flow. Joining two paths is what a merge is for. An empty `on error … { }` handler inside a branch whose sibling also returns gave one two: if … then $r = call microflow M.Sub() on error without rollback { }; return $r; -- the normal path reaches the end event else return 'no'; -- and so does this one end if; Measured on the one microflow that tripped it, FeedbackModule.SUB_Feedback_SendToServer: stored, two end events with in-degree 1 each and five merges; after the round trip, one end event with in-degree 2 and one merge. The colliding edges are the else branch's return and the error-handler flow out of "Post feedback to App Insights". Fixed as a post-pass over the finished graph, beside applyFlowCurves, not as a guard where the error flow is wired: the two flows come from unrelated builders in either order — here the error flow lands FIRST and the branch's flow arrives after — so neither creation site can see the collision. A first attempt at the wiring site compiled, ran, and changed nothing, which is what sent me looking. A merge and not a second end event, because that is what Studio Pro writes here and because the microflow has one return value that a second end event would need its own copy of. Scoped to end events: other nodes also take one inbound flow, but an end event is the only one measured to collide, and rewriting nodes with real inbound semantics on no evidence would be a guess. Whole-project round trip after the fix: 0 errors, from 1. The test parses real MDL through visitor.Build rather than hand-building the AST — the hand-built equivalent puts the two paths on separate end events and does not reproduce the bug at all, which is itself worth knowing. Controls: a handler with a statement in it, and a plain if/else, must gain no merge; stubbing the post-pass fails the first test with the reported symptom ("end event … has 2 incoming sequence flows, want 1"). Not fixed, and now recorded as a separate gap: DESCRIBE cannot say "the error flow rejoins the main path here", so a round trip still flattens Studio Pro's merge chain. That is a cosmetic diff rather than an unbuildable project, and closing it needs new MDL syntax. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../microflow-rewrite-property-audit.md | 45 ++++- ...icroflow-empty-error-handler-end-event.mdl | 72 +++++++ mdl/executor/cmd_microflows_builder_flows.go | 87 ++++++++ mdl/executor/cmd_microflows_builder_graph.go | 6 + .../cmd_microflows_end_event_merge_test.go | 186 ++++++++++++++++++ 6 files changed, 390 insertions(+), 7 deletions(-) create mode 100644 mdl-examples/bug-tests/microflow-empty-error-handler-end-event.mdl create mode 100644 mdl/executor/cmd_microflows_end_event_merge_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 57b690683..1416dd9f7 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -593,3 +593,4 @@ {"area": "mdl/executor", "date": "2026-09-12", "symptom": "Two unrelated validators shipped the same rule ID. `statictext` was given MDL-WIDGET27, which validate_widget_object_property.go already owned (a repeatable widget property written as a value) — and MDL-WIDGET28, the next number, was taken too, by a validator that exists on the default branch but not on the working branch", "cause": "The branch was 18 commits behind main. `git grep MDL-WIDGET27` on the WORKING TREE would have found the first collision and was not run; nothing at all would have found the second, because that rule is not in the branch. Rule IDs are allocated by picking the next unused number, which is a global allocation done against a local view", "file": "`mdl/executor/validate_widget_retired.go`, `mdl/executor/rule_id_uniqueness_test.go` (new)", "insight": "**A rule ID is allocated against the DEFAULT BRANCH, not the working tree** — `git grep origin/main` before claiming a number, and `git grep -ho 'MDL0[0-9][0-9]' origin/main | sort -u | tail` to find the real high-water mark. A branch behind by a few commits sees a free number that is not. **The invariant to test is one OWNER, not one occurrence**: a rule legitimately fires from several branches of its own validator (MDL-WIDGET25 does), so counting occurrences is a false alarm generator; counting distinct FILES is the useful proxy. Adding that guard immediately found a second collision from earlier the same day — MDL059 reused for a document-level annotation — which on inspection was a genuine extension of one rule to a second site (someone suppressing MDL059 means both), so it went on the exemption list with that reason rather than being renumbered. **Renumbering a shipped rule is user-visible** (it appears in output, --format json/sarif, and suppressions), which is why an existing overlap gets recorded rather than fixed. The guard cannot see a rule that is not in the repository, and says so in its own comment — that limit is the other half of the same bug", "refs": []} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget action slot given a real keyword short its argument (`Action: OPEN_LINK` with no URL) or an invented one (`Action: TOTALLY_MADE_UP`) was written as Forms$NoAction: a control that rendered, carried its caption, and did nothing. `mxcli check` passed, `exec` said \"Created page\", `mx check` gave 0 errors, `describe page` showed no action at all", "cause": "actionExprV3 spells `OPEN_LINK STRING_LITERAL`, so a bare OPEN_LINK cannot match it — and ANTLR does not fail, it falls through to the generic `keyword COLON propertyValueV3` alternative at the end of widgetPropertyV3. The slot then holds a plain string; WidgetV3.GetAction type-asserts to *ast.ActionV3, gets nil, and the writer's default branch emits Forms$NoAction. Every under-specified form went the same way (SHOW_PAGE, CREATE_OBJECT, COMPLETE_TASK, MICROFLOW, NANOFLOW), in Action:, OnClick: and OnChange: alike", "file": "`mdl/grammar/domains/MDLPage.g4` (actionExprV3, NOTHING promoted), `mdl/visitor/visitor_page_v3.go` (buildActionV3), `mdl/executor/cmd_pages_builder_v3.go` (buildClientActionV3 case \"none\"), `mdl/executor/validate_widget_action_slot.go` (new, MDL-WIDGET28)", "insight": "**Before rejecting a degraded form, check whether anything DOCUMENTED depends on the degradation.** `Action: NOTHING` — the shipped spelling for a deliberately inert widget, in the docs site, MDL_QUICK_REFERENCE, the synced alter-page skill and nine mdl-examples scripts — was never in the grammar: it reached Forms$NoAction through the very fall-through the report is about. The reporter's own proposal (\"make it a parse error\") would therefore have broken working syntax, and a naive `any scalar in an action slot is an error` rule breaks the example corpus on the first run. The measurement that settled it took one command — sweep every action value in mdl-examples and count them (`nothing` was the only non-grammar scalar, 9 uses) — and it should come BEFORE writing the rule, not after `make check-mdl` fails. The fix is then two coupled halves: promote the documented form to a real grammar alternative, and only then report the rest. Two controls prove the coupling: stub the validator and 7 tests fail with the reported symptom; delete the grammar alternative and the VISITOR NO LONGER COMPILES (`actCtx.NOTHING undefined`), which is a harder coupling than any test. Also: the builder's action switch ends in a `default:` that refuses unknown types, so promoting a grammar form without adding its case turns a working spelling into an exec failure — grammar, visitor and builder move together or not at all. Third appearance of this class: SIGN_OUT and OPEN_LINK both used to reach Forms$NoAction through the WRITER's default branch (CapTrackV2 FINDINGS §10); this is the first at the parser layer, and the general shape is that a silent-degrade path with a legal-looking destination is invisible to every downstream check, because the destination really is legal.", "refs": ["mendixlabs/mxcli#1062"]} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "An XPath constraint naming an association BARE \u2014 `[Ticket_Reporter = $currentUser]` \u2014 passes `mxcli check --references`, `exec` reports success, and the build then fails with `Error(s) in XPath constraint` (CE0161) naming the retrieve activity. Because `exec` applies statements one at a time and cannot roll back, a script carrying this in the middle leaves the model half-updated", "cause": "An association in an XPath constraint must be QUALIFIED (`Module.Assoc`); an attribute is bare. Nothing checked it. The fix (MDL-XPATH01, `mdl/executor/validate_xpath_association.go`) flags a bare name that is NOT an attribute of the constrained entity AND IS a known association, so it can name the spelling to use. Measured on a blank Mendix 11.14.0 app with qualification as the only variable: unqualified \u2192 check passed / build exit 3; qualified \u2192 check passed / BUILD SUCCEEDED", "file": "`mdl/executor/validate_xpath_association.go`, wired from `validateFlowBodyReferences` in `validate.go`; script-declared associations collected by `scriptContext.recordAssociation`", "insight": "**`scriptContext` has TWO parallel collectors over the same statement types \u2014 `collectDefinitions` (whole program) and `collectSingle` (incremental) \u2014 and a case added to one only is silently half-collected.** That is exactly how this rule first shipped: it fired against STORED associations and stayed silent on script-declared ones, which is the majority shape (one script creating the entity, the association and the microflow together) and the one that reaches a build half-written. Unit tests on the matcher were all green, because the rule was correct and simply never given the data \u2014 the miss only showed up running the real `.mdl` through `make check-mdl`. Both collectors now call one `recordAssociation`/`recordEntityAttrs` helper, and `TestBothCollectorsRecordAssociationsAndAttrs` asserts they agree (control: reverting the `collectDefinitions` case fails it with `associations[X] = \"\"`). Second lesson: **a rule that needs a project cannot be tested by a `.fail.mdl`** \u2014 `make check-mdl` runs `mxcli check` with NO project, so such a file passes silently and asserts nothing. The negative case belongs in Go; keep the `-ok.mdl` as the positive control so the rule cannot degrade to rejecting every constraint", "refs": ["ako/ChipCoV1 FINDINGS.md"]} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "`describe microflow` -> `exec` across every microflow in a project produced one that mxbuild refuses: CE0709 \"Sequence flow is not accepted by origin or destination\", with no microflow named in the message. `mxcli check` passed, `exec` reported success, and the project still OPENED in Studio Pro — only a build caught it. 1 of 41 microflows in ako/TestApp", "cause": "A Mendix end event accepts exactly ONE incoming sequence flow — joining two paths is what a merge is for. An empty `on error … { }` handler inside a branch whose sibling also returns gave one two: the error-handler flow and the else branch's return both landed on the same end event. DESCRIBE emits the empty block because MDL cannot say \"the error flow rejoins the main path at a merge\", which is what the stored Studio Pro document does", "file": "`mdl/executor/cmd_microflows_builder_flows.go` (mergeOverConnectedEndEvents), called from `cmd_microflows_builder_graph.go` beside applyFlowCurves", "insight": "**Measure in/out degree per node when a graph-shaped document fails to build.** CE0709 names neither the microflow nor the edge, and the round-trip diff was ~90% layout, so reading it got nowhere; counting inbound flows per node found the single over-connected end event in one pass and named the two colliding edges. **The guard belongs in a post-pass, not at the creation site**: the two flows are emitted by unrelated builders in either order — here the error flow lands FIRST and the branch's flow arrives afterwards — so a check at either site sees nothing wrong. A first attempt at the wiring site built cleanly and changed nothing, which is the tell. `applyFlowCurves` was already there for the same reason and is the precedent to copy. **Hand-built AST did not reproduce it** — the synthetic version put the two paths on separate end events — so the test parses real MDL through `visitor.Build`; when a builder bug will not reproduce from a hand-made AST, that is evidence the visitor's output differs, not that the bug is imaginary. **Prefer the join Studio Pro writes**: a second end event would also satisfy the in-degree rule but needs its own return value, which MDL never stated. The two microflows that lose merges and still build clean are the control that in-degree 2, not merge loss, is the trigger", "refs": []} diff --git a/docs/12-bug-reports/microflow-rewrite-property-audit.md b/docs/12-bug-reports/microflow-rewrite-property-audit.md index 0901c2b9d..c7b4b9cec 100644 --- a/docs/12-bug-reports/microflow-rewrite-property-audit.md +++ b/docs/12-bug-reports/microflow-rewrite-property-audit.md @@ -124,13 +124,44 @@ as an ordinary identifier — pinned by a test. ### Still open, and newly measured -Round-tripping **every** microflow in TestApp produces a project that does not -build: **CE0709** "Sequence flow is not accepted by origin or destination". The -control is the same operation on the pre-fix binary, which gives the identical -error — so this is pre-existing flow-graph drift, not a consequence of these -fixes, and it belongs to the "flow graph changed in 40/42" row rather than to -either property above. It is a stronger statement of the reviewability problem -than the 417-line diff, and wants its own investigation. +~~Round-tripping **every** microflow in TestApp produces a project that does not +build: **CE0709** "Sequence flow is not accepted by origin or destination".~~ +**Diagnosed and fixed.** It was never "flow-graph drift" — that was a label for +an unexplained symptom, and the "flow graph changed in 40/42" row it was filed +under is mostly bezier vectors and connection indices, so it pointed nowhere. + +In Mendix an end event accepts exactly **one** incoming sequence flow; joining +two paths is what a merge is for. An empty `on error … { }` handler inside a +branch whose sibling also returns gave one two. Measured on the single microflow +that tripped it, `FeedbackModule.SUB_Feedback_SendToServer`: + +| | stored (Studio Pro) | after the round trip | +|---|---|---| +| end events | two, in-degree 1 each | one in-degree 1, one **in-degree 2** | +| exclusive merges | 5 | 1 | + +The two colliding flows are the `else` branch's return and the error-handler flow +out of `Post feedback to App Insights`. Reproduced in six lines +(`mdl-examples/bug-tests/microflow-empty-error-handler-end-event.mdl`): the same +microflow with a statement in the handler block builds clean. + +The fix is a post-pass over the finished graph — an end event two paths reach +gets a merge in front of it — rather than a guard at the site that wires the +error flow, because the two flows are created by unrelated builders in either +order and neither site can see the collision. A merge and not a second end event: +that is what Studio Pro writes, and the microflow has one return value. + +Whole-project round trip after the fix: **0 errors**, from 1. + +Two of the three microflows that lose merges (`ConvertBase64String`, +`VAL_Feedback`) always built clean, because their freed branches each got their +own end event — which is the control that makes "in-degree 2 is the trigger" a +measurement rather than a story. + +Still open, and separate: **DESCRIBE cannot spell "the error flow rejoins the +main path here"**, so a round trip still flattens Studio Pro's merge chain. That +is now a cosmetic diff rather than an unbuildable project, and closing it needs +new MDL syntax. `ConcurrenyErrorMessage` and `ConcurrencyErrorMicroflow` are still hardcoded. Neither is a demonstrated loss (see the benign table), so closing those holes diff --git a/mdl-examples/bug-tests/microflow-empty-error-handler-end-event.mdl b/mdl-examples/bug-tests/microflow-empty-error-handler-end-event.mdl new file mode 100644 index 000000000..aceda8e8e --- /dev/null +++ b/mdl-examples/bug-tests/microflow-empty-error-handler-end-event.mdl @@ -0,0 +1,72 @@ +-- An empty `on error … { }` gave an end event two incoming flows (CE0709). +-- +-- In Mendix an end event accepts exactly ONE incoming sequence flow — joining +-- two paths is what a merge is for. The shape below produced two: the THEN +-- branch's error-handler flow and the ELSE branch's return both landed on the +-- same end event, and mxbuild rejected it with +-- +-- [error] [CE0709] "Sequence flow is not accepted by origin or destination." +-- +-- `mxcli check` passed, `exec` wrote it, and the project still OPENED — the +-- document is well formed, just over-connected — so only a build caught it. +-- That is how it survived a describe → exec round trip of all 41 microflows in +-- ako/TestApp with every other check green: 1 microflow tripped it, +-- FeedbackModule.SUB_Feedback_SendToServer, whose stored Studio Pro version +-- routes the error flow through a chain of merges that DESCRIBE cannot spell +-- and therefore emits as the empty handler block below. +-- +-- The fix is a post-pass over the finished graph: an end event that two paths +-- reach gets an exclusive merge in front of it. A merge, not a second end event +-- — Studio Pro writes a merge here, and the microflow has one return value, +-- which a second end event would need its own copy of. +-- +-- The DESCRIBE side is a separate gap and is NOT fixed: MDL still has no way to +-- say "the error flow rejoins the main path here", so a round trip flattens the +-- merge chain. That is now harmless rather than unbuildable. + +create or modify module EhProbe; + +create or replace microflow EhProbe.Helper () returns String +begin + return 'x'; +end; +/ + +-- THE CASE. Both branches return, and the THEN branch's call carries an empty +-- error handler. Before the fix this built a project mxbuild refused. +create or replace microflow EhProbe.EmptyHandler () returns String +begin + if 1 = 1 then + $r = call microflow EhProbe.Helper() on error without rollback { }; + return $r; + else + return 'no'; + end if; +end; +/ + +-- CONTROL 1: the same shape with a statement in the handler block. It routes +-- its own path, never needed the merge, and must not gain one. Without this the +-- file would pass equally against a builder that merged in front of every end +-- event unconditionally. +create or replace microflow EhProbe.FilledHandler () returns String +begin + if 1 = 1 then + $r = call microflow EhProbe.Helper() on error without rollback { return 'err'; }; + return $r; + else + return 'no'; + end if; +end; +/ + +-- CONTROL 2: no error handling at all. The post-pass must leave an ordinary +-- if/else alone — the narrowest statement that it does not rewrite every graph. +create or replace microflow EhProbe.PlainIf () returns String +begin + if 1 = 1 then + return 'yes'; + else + return 'no'; + end if; +end; diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 330b288d4..31273e361 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -1027,3 +1027,90 @@ func containsTerminalStmt(stmts []ast.MicroflowStatement) bool { } return false } + +// mergeOverConnectedEndEvents gives every end event that more than one path +// reaches an exclusive merge to join them at. +// +// An end event accepts exactly ONE incoming sequence flow — joining two paths +// is what a merge is for — so a second flow into one is CE0709 "Sequence flow +// is not accepted by origin or destination". Only mxbuild catches it: the +// document is otherwise well formed, so `mxcli check` passes and the project +// still opens, which is how this survived a describe → exec round trip of a +// whole project with everything else green. +// +// The shape that produced it is an empty `on error … { }` handler inside a +// branch whose sibling also returns: +// +// if … then +// $r = call microflow M.Sub() on error without rollback { }; +// return $r; -- the normal path reaches the end event +// else +// return 'no'; -- and so does this one +// end if; +// +// It runs as a post-pass rather than at the site that wires the error flow, +// because the two colliding flows are created by unrelated builders in either +// order: the error flow lands first and the branch's flow arrives afterwards, +// so neither site can see the collision. Same reasoning as applyFlowCurves. +// +// Merging is what Studio Pro writes here, and it is also what keeps the +// microflow's single return value — a second end event would need one of its +// own, and MDL never said what it should be. +// +// Scoped to end events on purpose. Other node types also accept one inbound +// flow, so the same collision is possible in principle, but an end event is the +// only one measured to occur (1 microflow in 41 across the audit corpus) and +// widening the rewrite to nodes with real inbound semantics — a loop, a merge's +// own feeders — without a case to test it on would be a guess. +func (fb *flowBuilder) mergeOverConnectedEndEvents() { + endEvents := make(map[model.ID]bool) + for _, obj := range fb.objects { + if ev, ok := obj.(*microflows.EndEvent); ok { + endEvents[ev.ID] = true + } + } + if len(endEvents) == 0 { + return + } + + // First-seen order, so the merges are created deterministically rather than + // in map order — a shuffled object list is a spurious diff on every write. + var order []model.ID + inbound := make(map[model.ID][]int) + for i, flow := range fb.flows { + if flow == nil || !endEvents[flow.DestinationID] { + continue + } + if _, seen := inbound[flow.DestinationID]; !seen { + order = append(order, flow.DestinationID) + } + inbound[flow.DestinationID] = append(inbound[flow.DestinationID], i) + } + + for _, endID := range order { + idxs := inbound[endID] + if len(idxs) < 2 { + continue + } + merge := µflows.ExclusiveMerge{ + BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Position: model.Point{X: fb.posX - HorizontalSpacing/2, Y: fb.baseY}, + Size: model.Size{Width: MergeSize, Height: MergeSize}, + }, + } + fb.objects = append(fb.objects, merge) + + // The merge inherits how the first path entered the end event; the + // re-pointed flows lose it, because a connection index describes an + // edge's landing on a node and these now land on the merge. + destIndex := fb.flows[idxs[0]].DestinationConnectionIndex + for _, i := range idxs { + fb.flows[i].DestinationID = merge.ID + fb.flows[i].DestinationConnectionIndex = 0 + } + mergeFlow := newHorizontalFlow(merge.ID, endID) + mergeFlow.DestinationConnectionIndex = destIndex + fb.flows = append(fb.flows, mergeFlow) + } +} diff --git a/mdl/executor/cmd_microflows_builder_graph.go b/mdl/executor/cmd_microflows_builder_graph.go index cff0617eb..48f748928 100644 --- a/mdl/executor/cmd_microflows_builder_graph.go +++ b/mdl/executor/cmd_microflows_builder_graph.go @@ -204,6 +204,12 @@ func (fb *flowBuilder) buildFlowGraph(stmts []ast.MicroflowStatement, returns *a // one. (#884) fb.applyFlowCurves() + // Give any end event that two paths reach a merge to join them at. Also + // here, and for the same reason as the curves: the two flows can be created + // by unrelated builders in either order, so no single creation site can see + // the collision. + fb.mergeOverConnectedEndEvents() + return µflows.MicroflowObjectCollection{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, Objects: fb.objects, diff --git a/mdl/executor/cmd_microflows_end_event_merge_test.go b/mdl/executor/cmd_microflows_end_event_merge_test.go new file mode 100644 index 000000000..9fefafeae --- /dev/null +++ b/mdl/executor/cmd_microflows_end_event_merge_test.go @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// An end event accepts exactly ONE incoming sequence flow — joining two paths +// is what a merge is for — and an empty `on error … { }` handler inside a branch +// whose sibling also returns gave one two, which mxbuild rejects with CE0709 +// "Sequence flow is not accepted by origin or destination". +// +// Only mxbuild caught it. The document is otherwise well formed, so `mxcli +// check` reported success, `exec` wrote it, and the project still opened — +// which is how it survived a describe → exec round trip of every microflow in +// ako/TestApp with every other check green (1 of 41 microflows, +// FeedbackModule.SUB_Feedback_SendToServer, whose stored Studio Pro version +// routes the error flow through a chain of merges that DESCRIBE cannot spell +// and emits as an empty handler block). + +// endEventInDegree counts inbound sequence flows per end event, at the top level +// and inside any loop body. +func endEventInDegree(col *microflows.MicroflowObjectCollection) map[model.ID]int { + ends := map[model.ID]bool{} + var findEnds func(objs []microflows.MicroflowObject) + findEnds = func(objs []microflows.MicroflowObject) { + for _, o := range objs { + switch v := o.(type) { + case *microflows.EndEvent: + ends[v.ID] = true + case *microflows.LoopedActivity: + findEnds(v.ObjectCollection.Objects) + } + } + } + findEnds(col.Objects) + + in := map[model.ID]int{} + var count func(flows []*microflows.SequenceFlow) + count = func(flows []*microflows.SequenceFlow) { + for _, f := range flows { + if f != nil && ends[f.DestinationID] { + in[f.DestinationID]++ + } + } + } + count(col.Flows) + var walk func(objs []microflows.MicroflowObject) + walk = func(objs []microflows.MicroflowObject) { + for _, o := range objs { + if loop, ok := o.(*microflows.LoopedActivity); ok { + count(loop.ObjectCollection.Flows) + walk(loop.ObjectCollection.Objects) + } + } + } + walk(col.Objects) + return in +} + +func assertNoOverConnectedEndEvent(t *testing.T, col *microflows.MicroflowObjectCollection) { + t.Helper() + seen := false + for id, n := range endEventInDegree(col) { + seen = true + if n > 1 { + t.Errorf("end event %s has %d incoming sequence flows, want 1 — "+ + "two paths must join at a merge (CE0709)", id, n) + } + } + if !seen { + t.Fatal("no end event in the graph; the assertion would be vacuous") + } +} + +// buildMicroflowFromMDL parses real MDL and builds its flow graph, so the test +// exercises the AST the visitor actually produces. Hand-building the equivalent +// AST does NOT reproduce this bug — the two paths end up at separate end events +// and never collide — which is itself the reason to go through the parser. +func buildMicroflowFromMDL(t *testing.T, src string) *microflows.MicroflowObjectCollection { + t.Helper() + prog := parseMDL(t, src) + var create *ast.CreateMicroflowStmt + for _, stmt := range prog.Statements { + if c, ok := stmt.(*ast.CreateMicroflowStmt); ok { + create = c + break + } + } + if create == nil { + t.Fatal("no CREATE MICROFLOW in the parsed program") + } + fb := &flowBuilder{ + posX: 100, + posY: 100, + spacing: HorizontalSpacing, + measurer: &layoutMeasurer{}, + varTypes: map[string]string{}, + } + return fb.buildFlowGraph(create.Body, create.ReturnType) +} + +// The shape that failed: a call with an EMPTY custom error handler in a branch +// that returns, beside an else branch that also returns. +const emptyHandlerMDL = `create microflow M.EmptyHandler () returns String +begin + if 1 = 1 then + $r = call microflow M.Sub() on error without rollback { }; + return $r; + else + return 'no'; + end if; +end;` + +const filledHandlerMDL = `create microflow M.FilledHandler () returns String +begin + if 1 = 1 then + $r = call microflow M.Sub() on error without rollback { return 'err'; }; + return $r; + else + return 'no'; + end if; +end;` + +const plainIfMDL = `create microflow M.PlainIf () returns String +begin + if 1 = 1 then + return 'yes'; + else + return 'no'; + end if; +end;` + +func TestBuilder_EmptyErrorHandlerDoesNotOverConnectEndEvent(t *testing.T) { + col := buildMicroflowFromMDL(t, emptyHandlerMDL) + assertNoOverConnectedEndEvent(t, col) + + // The join has to be a merge, not a second end event: the microflow has one + // return value and MDL never said what a second one would return. + merges := 0 + for _, o := range col.Objects { + if _, ok := o.(*microflows.ExclusiveMerge); ok { + merges++ + } + } + if merges == 0 { + t.Error("no ExclusiveMerge was created to join the two paths") + } +} + +// The control. Without it this would pass just as well against a builder that +// emitted a merge in front of every end event unconditionally, and against one +// that dropped the error flow entirely. +func TestBuilder_FilledErrorHandlerIsUnchanged(t *testing.T) { + col := buildMicroflowFromMDL(t, filledHandlerMDL) + assertNoOverConnectedEndEvent(t, col) + + // A handler with a body routes its own path and never needed the post-pass; + // this pins that it did not gain a merge it does not use. + errFlows := 0 + for _, f := range col.Flows { + if f != nil && f.IsErrorHandler { + errFlows++ + } + } + if errFlows != 1 { + t.Errorf("error-handler flows = %d, want 1", errFlows) + } +} + +// A microflow with no error handling at all must be untouched by the post-pass — +// the narrowest statement that it does not rewrite ordinary graphs. +func TestBuilder_PlainIfGainsNoMerge(t *testing.T) { + col := buildMicroflowFromMDL(t, plainIfMDL) + assertNoOverConnectedEndEvent(t, col) + for _, o := range col.Objects { + if _, ok := o.(*microflows.ExclusiveMerge); ok { + t.Error("a plain if/else with no error handling gained an ExclusiveMerge") + } + } +} From f1af1a5a786a83b5648c5466c890c299e2f4bada Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 20:47:09 +0000 Subject: [PATCH 14/18] docs(proposal): measure MDL-FLOW01 prevalence, which selects Mode 3 Phases 1 and 2 were gated on a prevalence scan that had never been run, so the proposal had sat at "unscheduled" since 2026-08-20 with its own outcome table unable to fire. Scanned 555 microflows across ten Mendix-authored Marketplace modules installed into one 11.6.6 project. Marketplace code rather than demo projects, because a content id and a version reproduce the corpus exactly where two numbers pasted into a doc do not. All 44 findings are committed as data so the table can be recomputed instead of trusted. 5.8% of all microflows are irreducible, 12.5% of the ones that branch, in 7 of 10 modules -- and 80% of the findings are recombinable. That is the table's middle row: Mode 3 earns its cost, with Mode 2 the honest fallback for the ~20% that stay crossed. Scheduling stays a separate call. Two limits recorded with the numbers, both of which bound the claim: LintContext.Microflows filters through notPlatformModule, so the shipped lint rule never sees this corpus at all -- the coverage that does reach it is the describe-time warning, and prevalence in a user's own modules is still unmeasured. FullMicroflow also returns nothing for RULE- and NANOFLOW-typed flows, so rules are outside the detector's reach despite being able to branch. Control, since a rule that never runs and a rule that finds nothing look identical: the scan reported what it examined (seen=599 loadfail=44 analysed=555), and the 44 failures reconcile exactly with the catalog's 39 NANOFLOW + 5 RULE rows. Discrimination is shown by a pair -- Administration.ManageMyAccount with one branching split is flagged and warns on describe, Email_Connector.VAL_EmailTemplateRecipients with ten is silent on both -- so the detector keys on structure, not on branch volume. Also corrects the scan command: the flag is -r/--rules, not --rule, and status drops from draft to partial now that Phase 0 is merged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- ...OPOSAL_structured_microflow_description.md | 94 ++++- .../data/flow01-prevalence-2026-09-12.json | 354 ++++++++++++++++++ 2 files changed, 441 insertions(+), 7 deletions(-) create mode 100644 docs/11-proposals/data/flow01-prevalence-2026-09-12.json diff --git a/docs/11-proposals/PROPOSAL_structured_microflow_description.md b/docs/11-proposals/PROPOSAL_structured_microflow_description.md index a03810243..80eaffeb1 100644 --- a/docs/11-proposals/PROPOSAL_structured_microflow_description.md +++ b/docs/11-proposals/PROPOSAL_structured_microflow_description.md @@ -1,13 +1,15 @@ --- title: Structured description of irreducible microflow graphs -status: draft +status: partial date: 2026-08-20 --- # Proposal: Structured description of irreducible microflow graphs -**Status:** Draft -**Date:** 2026-08-20 +**Status:** Partial — Phase 0 (detector, `MDL-FLOW01`, describe-time warning) is +shipped; the prevalence scan that gates the rest is +[measured below](#measured-2026-09-12) and selects Mode 3. Phases 1–2 unscheduled. +**Date:** 2026-08-20 (scan: 2026-09-12) `DESCRIBE MICROFLOW` renders a microflow's control flow as nested `if/then/else`. That works only for graphs that are *properly nested*. A Mendix microflow is an @@ -253,10 +255,10 @@ Classification for each irreducible split: - **interleaved** — the intersection contains an activity, or the branches have more than one shared entry point. -### The prevalence scan (to be run against demo projects) +### The prevalence scan ```bash -mxcli lint -p app.mpr --rule MDL-FLOW01 --format json +mxcli lint -p app.mpr -r MDL-FLOW01 --format json ``` Emit per finding: qualified microflow name, module, split position, @@ -265,10 +267,88 @@ classification, branch count, size of the overlap region. What the numbers decid | Result | Consequence | |---|---| | Irreducible graphs are rare | Ship Mode 2 only; refuse the rest. Mode 3 not worth building. | -| Common and mostly *recombinable* | Mode 3 earns its cost; it is the pretty answer for most of them. | +| **Common and mostly *recombinable*** | **Mode 3 earns its cost; it is the pretty answer for most of them.** | | Common and mostly *interleaved* | Mode 2 is the whole feature; Mode 3 would rarely apply. | -Until this is measured, Modes 2 and 3 are **unscheduled**. +#### Measured, 2026-09-12 + +Corpus: **555 microflows** across **10 Mendix-authored Marketplace modules** +installed into one 11.6.6 project. Administration 4.3.2 and FeedbackModule 4.0.2 +ship with a blank app; the rest were installed with +`mxcli marketplace install -p app.mpr`: + +| Module | Content id | Version | +|---|---:|---| +| Workflow Commons | 117066 | 4.5.0 (newest built for 11.6.6) | +| Email Connector | 120739 | 6.4.3 | +| DatabaseReplication | 160 | 9.3.1 | +| ExcelImporter | 72 | 11.2.2 | +| Encryption | 1011 | 11.1.2 | +| Audittrail | 138 | 10.2.2 | +| Community Commons | 170 | 11.5.1 | + +Marketplace code was chosen +over demo projects because the result is **reproducible by anyone** from a +content id and a version, rather than resting on two numbers pasted into a doc. +All 44 findings — module, microflow, class, branch count, overlap size, split +position — are in +[`data/flow01-prevalence-2026-09-12.json`](data/flow01-prevalence-2026-09-12.json), +so the table below can be recomputed rather than taken on trust. + +| Module | Microflows | With a branching split | Flagged | recombinable | interleaved | +|---|---:|---:|---:|---:|---:| +| WorkflowCommons | 176 | 63 | 3 | 3 | 0 | +| Email_Connector | 135 | 65 | 12 | 17 | 0 | +| DatabaseReplication | 121 | 72 | 8 | 7 | 6 | +| ExcelImporter | 79 | 37 | 6 | 4 | 3 | +| Encryption | 16 | 6 | 1 | 1 | 0 | +| AuditTrail | 8 | 3 | 0 | 0 | 0 | +| Administration | 8 | 4 | 1 | 1 | 0 | +| FeedbackModule | 7 | 3 | 1 | 2 | 0 | +| CommunityCommons | 4 | 4 | 0 | 0 | 0 | +| MyFirstModule | 1 | 0 | 0 | 0 | 0 | +| **Total** | **555** | **257** | **32** | **35** | **9** | + +- **5.8 %** of all microflows are irreducible (32 / 555), and **12.5 %** of the + ones that actually branch (32 / 257). **7 of 10** modules contain at least one. +- **44 findings** over those 32 microflows — a microflow can carry more than one + irreducible split. **80 % recombinable** (35), **20 % interleaved** (9). +- Interleaved graphs cluster: all 9 are in DatabaseReplication (6) and + ExcelImporter (3). Both are old modules — 9.3.1 and a long lineage — which is + consistent with crossed flows being something that accretes under maintenance + rather than something anyone draws on purpose. + +**This is the middle row.** Irreducible graphs are not a curiosity — one in eight +branching microflows written by Mendix's own teams cannot be described faithfully +today — and they are overwhelmingly *recombinable*, which is the class Mode 3 can +render as ordinary nested `if`s. So **Mode 3 earns its cost**, and Mode 2 is the +honest fallback for the ~20 % that stay crossed. Scheduling is still a separate +call; what is settled is that Mode 3 is not speculative work. + +Two caveats that matter more than the percentages: + +- **The shipped lint rule never sees this corpus.** `LintContext.Microflows()` + filters through `notPlatformModule`, so `mxcli lint` deliberately skips + Marketplace and System modules — linting code the user cannot edit would be + noise. The numbers above were obtained by bypassing that filter in a + throwaway build. The coverage that *does* reach these microflows is the + **describe-time warning**, which is not lint: describing + `Administration.ManageMyAccount` emits the #923 warning today. Prevalence in a + user's own modules is therefore still unmeasured, and may differ. +- **Rules are not covered.** `FullMicroflow` returns nothing for the 5 + `RULE`-typed flows (and the 39 nanoflows) in this project, so the rule skips + them silently. A rule is "a special kind of microflow" and can branch, so this + is a real gap in the detector's reach, not just in this measurement. + +Method and control, since a rule that never runs and a rule that finds nothing +look identical: the scan was instrumented to report what it actually examined — +`seen=599 loadfail=44 analysed=555 atrisk=257 branchingsplits=560 findings=44` — +and the 44 load failures reconcile exactly with the catalog's 39 `NANOFLOW` + 5 +`RULE` rows, so all 555 microflows were genuinely walked. The discriminating +control is a pair: `Administration.ManageMyAccount`, with **one** branching +split, is flagged and warns on describe, while +`Email_Connector.VAL_EmailTemplateRecipients`, with **ten**, is silent on both. +The detector keys on branch *structure*, not on how much a microflow branches. ### Files to modify/create diff --git a/docs/11-proposals/data/flow01-prevalence-2026-09-12.json b/docs/11-proposals/data/flow01-prevalence-2026-09-12.json new file mode 100644 index 000000000..757ca9da6 --- /dev/null +++ b/docs/11-proposals/data/flow01-prevalence-2026-09-12.json @@ -0,0 +1,354 @@ +[ + { + "module": "Administration", + "microflow": "ManageMyAccount", + "class": "recombinable", + "branches": 3, + "overlap": 3, + "split_at": "220, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "BCo_CustomConstraint", + "class": "recombinable", + "branches": 2, + "overlap": 2, + "split_at": "-365, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "BCo_CustomConstraint", + "class": "interleaved", + "branches": 3, + "overlap": 4, + "split_at": "-85, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "BCo_CustomConstraint", + "class": "interleaved", + "branches": 2, + "overlap": 4, + "split_at": "400, 65" + }, + { + "module": "DatabaseReplication", + "microflow": "BCo_CustomConstraint", + "class": "interleaved", + "branches": 3, + "overlap": 5, + "split_at": "540, 65" + }, + { + "module": "DatabaseReplication", + "microflow": "BCo_TableMapping", + "class": "interleaved", + "branches": 5, + "overlap": 30, + "split_at": "50, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "Column_SetCorrectRefObjectType", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "520, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "IVK_CustomConstraint_Save", + "class": "interleaved", + "branches": 10, + "overlap": 6, + "split_at": "487, 390" + }, + { + "module": "DatabaseReplication", + "microflow": "IVK_SaveImportCall", + "class": "recombinable", + "branches": 3, + "overlap": 4, + "split_at": "185, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "ValidateColumnMapping", + "class": "recombinable", + "branches": 3, + "overlap": 14, + "split_at": "-130, 350" + }, + { + "module": "DatabaseReplication", + "microflow": "ValidateColumnMapping", + "class": "recombinable", + "branches": 2, + "overlap": 5, + "split_at": "1355, 255" + }, + { + "module": "DatabaseReplication", + "microflow": "ValidateDatabase", + "class": "recombinable", + "branches": 2, + "overlap": 2, + "split_at": "265, 200" + }, + { + "module": "DatabaseReplication", + "microflow": "ValidateTableMapping", + "class": "recombinable", + "branches": 4, + "overlap": 2, + "split_at": "190, 230" + }, + { + "module": "DatabaseReplication", + "microflow": "ValidateTableMapping", + "class": "interleaved", + "branches": 5, + "overlap": 10, + "split_at": "50, 115" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_ClientCredentialsGrant_SaveAutoConfig", + "class": "recombinable", + "branches": 2, + "overlap": 22, + "split_at": "-1955, -415" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_ClientCredentialsGrant_SaveManualConfig", + "class": "recombinable", + "branches": 2, + "overlap": 12, + "split_at": "-1775, -290" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_GetOrRenewTokenJavaAction", + "class": "recombinable", + "branches": 2, + "overlap": 16, + "split_at": "-1760, -485" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_GetOrRenewTokenJavaAction", + "class": "recombinable", + "branches": 3, + "overlap": 2, + "split_at": "-770, -485" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveAutoConfig", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "-145, -240" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveAutoConfig", + "class": "recombinable", + "branches": 2, + "overlap": 20, + "split_at": "-908, -240" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveEmailSecurityConfiguration", + "class": "recombinable", + "branches": 2, + "overlap": 12, + "split_at": "-1060, -60" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveEmailSecurityConfiguration", + "class": "recombinable", + "branches": 2, + "overlap": 4, + "split_at": "-550, -60" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveManualConfig", + "class": "recombinable", + "branches": 2, + "overlap": 17, + "split_at": "-1545, -230" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_SaveManualConfig", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "-577, -230" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_Save_EmailSecurityConfig", + "class": "recombinable", + "branches": 2, + "overlap": 12, + "split_at": "-1089, -60" + }, + { + "module": "Email_Connector", + "microflow": "ACT_EmailAccount_Save_EmailSecurityConfig", + "class": "recombinable", + "branches": 2, + "overlap": 4, + "split_at": "-579, -60" + }, + { + "module": "Email_Connector", + "microflow": "OCH_LDapConfiguration_AuthType", + "class": "recombinable", + "branches": 2, + "overlap": 3, + "split_at": "25, 170" + }, + { + "module": "Email_Connector", + "microflow": "SUB_EmailAccount_Save", + "class": "recombinable", + "branches": 2, + "overlap": 13, + "split_at": "-2280, 75" + }, + { + "module": "Email_Connector", + "microflow": "SUB_EmailAccount_SaveReceiveConfig", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "-1638, -240" + }, + { + "module": "Email_Connector", + "microflow": "SUB_EmailAccount_SaveSendConfig", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "-1638, -240" + }, + { + "module": "Email_Connector", + "microflow": "VAL_BatchDetails", + "class": "recombinable", + "branches": 2, + "overlap": 1, + "split_at": "-385, 55" + }, + { + "module": "Encryption", + "microflow": "MB_SaveCertificate", + "class": "recombinable", + "branches": 3, + "overlap": 10, + "split_at": "-585, 200" + }, + { + "module": "ExcelImporter", + "microflow": "BCo_Column", + "class": "recombinable", + "branches": 12, + "overlap": 12, + "split_at": "770, 200" + }, + { + "module": "ExcelImporter", + "microflow": "Column_SetCorrectRefObjectType", + "class": "recombinable", + "branches": 2, + "overlap": 8, + "split_at": "520, 200" + }, + { + "module": "ExcelImporter", + "microflow": "Column_SetDetails", + "class": "recombinable", + "branches": 4, + "overlap": 2, + "split_at": "190, 200" + }, + { + "module": "ExcelImporter", + "microflow": "IVK_Column_Save", + "class": "interleaved", + "branches": 5, + "overlap": 12, + "split_at": "55, 200" + }, + { + "module": "ExcelImporter", + "microflow": "SetColumnStatus", + "class": "interleaved", + "branches": 12, + "overlap": 10, + "split_at": "335, 200" + }, + { + "module": "ExcelImporter", + "microflow": "SetColumnStatus", + "class": "recombinable", + "branches": 5, + "overlap": 3, + "split_at": "725, 200" + }, + { + "module": "ExcelImporter", + "microflow": "ValidateTemplate", + "class": "interleaved", + "branches": 5, + "overlap": 18, + "split_at": "1295, 230" + }, + { + "module": "FeedbackModule", + "microflow": "VAL_Feedback", + "class": "recombinable", + "branches": 2, + "overlap": 3, + "split_at": "-215, 200" + }, + { + "module": "FeedbackModule", + "microflow": "VAL_Feedback", + "class": "recombinable", + "branches": 2, + "overlap": 3, + "split_at": "770, 200" + }, + { + "module": "WorkflowCommons", + "microflow": "DS_WorkflowTaskDefinition_Selectable_UserImplementation", + "class": "recombinable", + "branches": 2, + "overlap": 5, + "split_at": "465, 170" + }, + { + "module": "WorkflowCommons", + "microflow": "OCH_DashboardContext_UpdateTaskDashboard", + "class": "recombinable", + "branches": 2, + "overlap": 2, + "split_at": "320, 200" + }, + { + "module": "WorkflowCommons", + "microflow": "SUB_UserTask_Assign", + "class": "recombinable", + "branches": 2, + "overlap": 3, + "split_at": "-190, 125" + } +] \ No newline at end of file From 36367220020de3cb76648c965e42fcd5ca33291e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 18:55:38 +0000 Subject: [PATCH 15/18] fix(view-entity,widgets): read Mendix's own OQL clause order, and stop dropping a widget with nowhere to go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects from ako/view-entity-examples FINDINGS, §3 and §2. Both are the same shape: mxcli goes quiet on input it cannot place, so the only symptom is something missing later. §3 — a view entity's OQL in the FROM-first clause order Mendix OQL puts the select list in either position and the MDL grammar accepts both (oqlQueryTerm). extractSelectClause handled only select-first: it found SELECT and scanned FORWARD for FROM, so in `from … group by … select …` the terminator was behind the keyword and the scan ran off the end, returning "". That order is what Studio Pro stores, so it is what DESCRIBE ENTITY emits — describe → edit → check was the one loop that broke on this document type. The empty string did two things and only one was visible: inferOQLTypes reported "could not parse select clause", while MDL030, MDL072 and every MDL031 type rule — all behind `if selectClause != ""` — silently stopped running. A from-first view got no OQL checking at all. So the fix is both halves. extractSelectClause now reads either order, and an unreadable select clause is REPORTED rather than skipped, because this is a recurrence at the same function: bug 9b (2026-07-09) was a case-comparison slip with the identical symptom and the identical hidden half. Two widening traps are pinned by tests: the orders need different terminator sets (admitting ORDER/LIMIT in select-first cuts `select o.Limit as Limit from …` in half), and the scanner skips quoted runs, since `s."Order"` is the form mxcli tells users to write so a reserved word survives CE0174. §2 — a child a pluggable widget has nowhere to put The four passes that place a pluggable widget's children each `continue` past what they do not recognise, and applyChildSlots parks leftovers in defaultWidgets, which is assigned only if the widget declares a `template` slot. A Gallery has one; DataGrid 2 does not, so its leftovers were built and discarded — check silent, exec successful, build clean, and DESCRIBE PAGE showing the widget gone as the only symptom. The reported case needed two mistakes, and the docs supplied one: `filter` is the Gallery keyword for the filters placeholder (DataGrid 2 spells the same property `controlbar`), and `widgetV3` is `type name props? body?`, so a filter written after a column's parentheses is a sibling widget rather than a block on the column. `mxcli syntax page widgets` documented exactly that form, so the likeliest thing to write was the thing that vanished; it now shows the working one. MDL-WIDGET29 reports it at check time and buildPluggable refuses it at exec time, both from one predicate so they cannot disagree. The predicate is conservative — silent with no definition, with no child slots, or when a `template` catch-all exists — so check can never claim a drop exec would not make. The message names the spelling this widget uses for the same slot, derived from the table the engine already reads. Because that definition is read from the project's installed .mpk, upgrading Data Widgets from the Marketplace keeps the rule current with no mxcli release. Verification Both proven to be the cause by reverting them. §3: the clause-order tests fail with the reported "" and with `got []` where the rules had gone quiet. §2, measured end to end on Mendix 11.13.0 with DataGrid 2 (Data Widgets) 3.4.0 — with the guard stubbed, `exec --no-check` printed "Created page", exited 0, and `describe page` came back with the column and no filter at all, reproducing the report against a real model; with the guard in place the same script is refused naming `controlbar`, the column-braces form persists as `column … { textfilter tf }`, and `mx check` on the result is 0 errors. Controls throughout — each from-first query paired with its select-first twin, and the real Gallery definition, on which the same `filter { … }` block must stay valid. go test ./... clean; make lint clean; make check-mdl 521 pass, 0 fail; make check-findings OK. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 2 + .claude/skills/mendix/overview-pages/SKILL.md | 22 ++ .../skills/mendix/write-oql-queries/SKILL.md | 25 ++ cmd/mxcli/syntax/features_page.go | 11 +- .../bug-patterns/silent-property-drop.md | 23 ++ docs/01-project/MDL_QUICK_REFERENCE.md | 3 + ...datagrid-filter-block-dropped-silently.mdl | 96 ++++++++ .../view-entity-oql-from-first-alias.fail.mdl | 31 +++ ...iew-entity-oql-from-first-clause-order.mdl | 87 +++++++ mdl/executor/oql_type_inference.go | 196 ++++++++++++---- mdl/executor/oql_type_inference_test.go | 11 +- .../validate_oql_clause_order_test.go | 222 ++++++++++++++++++ mdl/executor/validate_widgets.go | 6 + mdl/executor/widget_engine.go | 8 + mdl/executor/widget_unrouted_children.go | 175 ++++++++++++++ mdl/executor/widget_unrouted_children_test.go | 200 ++++++++++++++++ 16 files changed, 1068 insertions(+), 50 deletions(-) create mode 100644 mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl create mode 100644 mdl-examples/bug-tests/view-entity-oql-from-first-alias.fail.mdl create mode 100644 mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl create mode 100644 mdl/executor/validate_oql_clause_order_test.go create mode 100644 mdl/executor/widget_unrouted_children.go create mode 100644 mdl/executor/widget_unrouted_children_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index b5e965235..ae0c373db 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -594,3 +594,5 @@ {"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget action slot given a real keyword short its argument (`Action: OPEN_LINK` with no URL) or an invented one (`Action: TOTALLY_MADE_UP`) was written as Forms$NoAction: a control that rendered, carried its caption, and did nothing. `mxcli check` passed, `exec` said \"Created page\", `mx check` gave 0 errors, `describe page` showed no action at all", "cause": "actionExprV3 spells `OPEN_LINK STRING_LITERAL`, so a bare OPEN_LINK cannot match it — and ANTLR does not fail, it falls through to the generic `keyword COLON propertyValueV3` alternative at the end of widgetPropertyV3. The slot then holds a plain string; WidgetV3.GetAction type-asserts to *ast.ActionV3, gets nil, and the writer's default branch emits Forms$NoAction. Every under-specified form went the same way (SHOW_PAGE, CREATE_OBJECT, COMPLETE_TASK, MICROFLOW, NANOFLOW), in Action:, OnClick: and OnChange: alike", "file": "`mdl/grammar/domains/MDLPage.g4` (actionExprV3, NOTHING promoted), `mdl/visitor/visitor_page_v3.go` (buildActionV3), `mdl/executor/cmd_pages_builder_v3.go` (buildClientActionV3 case \"none\"), `mdl/executor/validate_widget_action_slot.go` (new, MDL-WIDGET28)", "insight": "**Before rejecting a degraded form, check whether anything DOCUMENTED depends on the degradation.** `Action: NOTHING` — the shipped spelling for a deliberately inert widget, in the docs site, MDL_QUICK_REFERENCE, the synced alter-page skill and nine mdl-examples scripts — was never in the grammar: it reached Forms$NoAction through the very fall-through the report is about. The reporter's own proposal (\"make it a parse error\") would therefore have broken working syntax, and a naive `any scalar in an action slot is an error` rule breaks the example corpus on the first run. The measurement that settled it took one command — sweep every action value in mdl-examples and count them (`nothing` was the only non-grammar scalar, 9 uses) — and it should come BEFORE writing the rule, not after `make check-mdl` fails. The fix is then two coupled halves: promote the documented form to a real grammar alternative, and only then report the rest. Two controls prove the coupling: stub the validator and 7 tests fail with the reported symptom; delete the grammar alternative and the VISITOR NO LONGER COMPILES (`actCtx.NOTHING undefined`), which is a harder coupling than any test. Also: the builder's action switch ends in a `default:` that refuses unknown types, so promoting a grammar form without adding its case turns a working spelling into an exec failure — grammar, visitor and builder move together or not at all. Third appearance of this class: SIGN_OUT and OPEN_LINK both used to reach Forms$NoAction through the WRITER's default branch (CapTrackV2 FINDINGS §10); this is the first at the parser layer, and the general shape is that a silent-degrade path with a legal-looking destination is invisible to every downstream check, because the destination really is legal.", "refs": ["mendixlabs/mxcli#1062"]} {"area": "mdl/executor", "date": "2026-09-11", "symptom": "An XPath constraint naming an association BARE \u2014 `[Ticket_Reporter = $currentUser]` \u2014 passes `mxcli check --references`, `exec` reports success, and the build then fails with `Error(s) in XPath constraint` (CE0161) naming the retrieve activity. Because `exec` applies statements one at a time and cannot roll back, a script carrying this in the middle leaves the model half-updated", "cause": "An association in an XPath constraint must be QUALIFIED (`Module.Assoc`); an attribute is bare. Nothing checked it. The fix (MDL-XPATH01, `mdl/executor/validate_xpath_association.go`) flags a bare name that is NOT an attribute of the constrained entity AND IS a known association, so it can name the spelling to use. Measured on a blank Mendix 11.14.0 app with qualification as the only variable: unqualified \u2192 check passed / build exit 3; qualified \u2192 check passed / BUILD SUCCEEDED", "file": "`mdl/executor/validate_xpath_association.go`, wired from `validateFlowBodyReferences` in `validate.go`; script-declared associations collected by `scriptContext.recordAssociation`", "insight": "**`scriptContext` has TWO parallel collectors over the same statement types \u2014 `collectDefinitions` (whole program) and `collectSingle` (incremental) \u2014 and a case added to one only is silently half-collected.** That is exactly how this rule first shipped: it fired against STORED associations and stayed silent on script-declared ones, which is the majority shape (one script creating the entity, the association and the microflow together) and the one that reaches a build half-written. Unit tests on the matcher were all green, because the rule was correct and simply never given the data \u2014 the miss only showed up running the real `.mdl` through `make check-mdl`. Both collectors now call one `recordAssociation`/`recordEntityAttrs` helper, and `TestBothCollectorsRecordAssociationsAndAttrs` asserts they agree (control: reverting the `collectDefinitions` case fails it with `associations[X] = \"\"`). Second lesson: **a rule that needs a project cannot be tested by a `.fail.mdl`** \u2014 `make check-mdl` runs `mxcli check` with NO project, so such a file passes silently and asserts nothing. The negative case belongs in Go; keep the `-ok.mdl` as the positive control so the rule cannot degrade to rejecting every constraint", "refs": ["ako/ChipCoV1 FINDINGS.md"]} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "`describe microflow` -> `exec` across every microflow in a project produced one that mxbuild refuses: CE0709 \"Sequence flow is not accepted by origin or destination\", with no microflow named in the message. `mxcli check` passed, `exec` reported success, and the project still OPENED in Studio Pro — only a build caught it. 1 of 41 microflows in ako/TestApp", "cause": "A Mendix end event accepts exactly ONE incoming sequence flow — joining two paths is what a merge is for. An empty `on error … { }` handler inside a branch whose sibling also returns gave one two: the error-handler flow and the else branch's return both landed on the same end event. DESCRIBE emits the empty block because MDL cannot say \"the error flow rejoins the main path at a merge\", which is what the stored Studio Pro document does", "file": "`mdl/executor/cmd_microflows_builder_flows.go` (mergeOverConnectedEndEvents), called from `cmd_microflows_builder_graph.go` beside applyFlowCurves", "insight": "**Measure in/out degree per node when a graph-shaped document fails to build.** CE0709 names neither the microflow nor the edge, and the round-trip diff was ~90% layout, so reading it got nowhere; counting inbound flows per node found the single over-connected end event in one pass and named the two colliding edges. **The guard belongs in a post-pass, not at the creation site**: the two flows are emitted by unrelated builders in either order — here the error flow lands FIRST and the branch's flow arrives afterwards — so a check at either site sees nothing wrong. A first attempt at the wiring site built cleanly and changed nothing, which is the tell. `applyFlowCurves` was already there for the same reason and is the precedent to copy. **Hand-built AST did not reproduce it** — the synthetic version put the two paths on separate end events — so the test parses real MDL through `visitor.Build`; when a builder bug will not reproduce from a hand-made AST, that is evidence the visitor's output differs, not that the bug is imaginary. **Prefer the join Studio Pro writes**: a second end event would also satisfy the in-degree rule but needs its own return value, which MDL never stated. The two microflows that lose merges and still build clean are the control that in-degree 2, not merge loss, is the trigger", "refs": []} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "`check --references` / `exec` on a view entity reports `could not parse select clause from OQL query` for a query Mendix itself wrote — specifically anything in the FROM-first clause order (`from … group by … select …`), which is what Studio Pro stores and therefore what `DESCRIBE ENTITY` emits. Feeding a describe straight back to check fails, so describe → edit → exec is the one loop that breaks on this document type", "cause": "`extractSelectClause` found SELECT and scanned FORWARD for FROM to end the column list. In the from-first order the FROM is BEHIND the SELECT, so the scan ran off the end and returned \"\". The MDL grammar had accepted both orders all along (`oqlQueryTerm`, MDLCatalog.g4) — only the hand-written scanner had not", "file": "`mdl/executor/oql_type_inference.go` (`extractSelectClause` → `extractSelectClauseOK` + `topLevelKeywordIndex`; the new report in `ValidateOQLSyntax`)", "insight": "**An empty select clause is a checker-off switch, and that is the real finding.** The visible symptom was ONE bogus error from `inferOQLTypes`; the invisible half was that MDL030 (missing alias), MDL072 (quoted alias) and every MDL031 type rule sit behind `if selectClause != \"\"` and simply stopped running — a from-first view got NO OQL checking at all. So the fix is two things: read both orders, AND report an unreadable clause instead of skipping it, so a third clause shape cannot go quiet again. **This is a RECURRENCE at the same function**: bug 9b (2026-07-09, same symptom string, same double consequence) was a case-comparison slip making it return \"\" for every query. Same fault, different input shape, two months apart — when a function's failure mode is 'returns empty and everything downstream shrugs', fix the shrug, not just the input. Two traps in widening it: the two orders need DIFFERENT terminator sets (admitting ORDER/LIMIT in the select-first order cuts `select o.Limit as Limit from …` in half, so the order is decided by whether FROM precedes SELECT), and the scanner must skip quoted runs, because `select s.\"Order\" as OrderValue` is exactly the form mxcli tells users to write so a reserved word survives MxBuild (CE0174). `ORDER BY` is matched as a phrase, since a bare ORDER is an ordinary name. Repro `mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl` + the `.fail.mdl` sibling that proves MDL030 runs on this order; tests `validate_oql_clause_order_test.go`, each pairing the from-first query with its select-first twin as the control. Reported by ako/view-entity-examples FINDINGS §3", "rules": ["MDL030", "MDL031", "MDL072"], "refs": ["ako/view-entity-examples"]} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget written inside a pluggable widget's body is BUILT AND THROWN AWAY — `check` silent, `exec` reports success, the build is clean, and `DESCRIBE PAGE` showing the widget gone is the only symptom. Reported as a Data Grid 2 column filter written `COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (…) }` — the form `mxcli syntax page widgets` documented", "cause": "The four passes that place a pluggable widget's children — `applyChildSlots`, `applyObjectLists`, and two auto-discovery passes in `buildPluggable` — each `continue` past a child they do not recognise, and `applyChildSlots` parks the leftovers in `defaultWidgets`, which is assigned ONLY if the widget declares a `template` slot (`defaultSlotContainer`). A Gallery has one; DataGrid 2 does not, so its leftovers were discarded with no error. Not datagrid-specific: one `continue` per pass, shared by every pluggable widget", "file": "`mdl/executor/widget_unrouted_children.go` (new: `unroutedPluggableChildren`, `validateUnroutedChildren` = MDL-WIDGET29, `refuseUnroutedChildren`), wired in `mdl/executor/widget_engine.go` (`buildPluggable`) and `mdl/executor/validate_widgets.go`", "insight": "**Two independent mistakes were needed to hide this, and the docs supplied one of them.** `FILTER` is the GALLERY keyword for the filters placeholder — DataGrid 2 spells the same property `controlbar` (`widgetSlotKeywordOverrides`) — and `widgetV3` is `type name props? body?`, so a `filter` written AFTER the column's parentheses is not a block on the column at all but a SIBLING widget. The syntax topic showed exactly that, so the likeliest thing to write was the thing that vanished. **Derive the diagnostic's advice from the table the engine already reads**: `widgetSlotKeywordOverrides` knows both spellings of the same property, so the message can say 'on this widget that slot is spelled `controlbar`' rather than listing containers to hunt through. **Check and exec share one predicate** (`unroutedPluggableChildren`) with a test asserting they agree, because a checker and a writer disagreeing about a drop is how this started. The predicate is deliberately CONSERVATIVE — silent with no definition, with no child slots (the auto pass may still take the child), and when a `template` catch-all exists — so `check` can never claim a drop `exec` would not make. **Check MDL-WIDGET rule numbers before picking one**: there is no registry, 01–28 were taken, and the obvious next number collided with the repeatable-property rule. Sibling rule MDL-WIDGET26 covers the neighbouring case (a container KEYWORD in the same position); it could not report this one because a real widget resolves perfectly well on its own. Repro `mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl` — a plain `.mdl`, not `.fail.mdl`, because the rule needs the parent's definition and `make check-mdl` runs with no project (the Makefile's #891/#892 note). Tests `widget_unrouted_children_test.go`, control = the real Gallery definition, on which the same `filter { … }` block must stay valid. **Measured end to end on Mendix 11.13.0 with DataGrid 2 (Data Widgets) 3.4.0**, which matters because the earlier write-up called this unverifiable without a project — DataGrid 2 ships in every Mendix 11 app, so the real definition is always one `-p` away. With the guard STUBBED, `exec --no-check` printed `Created page` and exited 0 while `describe page` came back `column MeterCode (Attribute: MeterCode, Caption: 'Meter')` with no filter at all — the reported defect reproduced against a real model. With the guard in place the same script is refused (exit 1, naming `controlbar`), the column-braces form persists as `column … { textfilter tf }`, and `mx check` on the result is 0 errors. Reading the definition from the project's `.mpk` is also what keeps the rule current: upgrading Data Widgets from the Marketplace changes what the rule sees with no mxcli release. Reported by ako/view-entity-examples FINDINGS §2", "rules": ["MDL-WIDGET29"], "refs": ["ako/view-entity-examples"]} diff --git a/.claude/skills/mendix/overview-pages/SKILL.md b/.claude/skills/mendix/overview-pages/SKILL.md index dbfc4b484..96f9220e8 100644 --- a/.claude/skills/mendix/overview-pages/SKILL.md +++ b/.claude/skills/mendix/overview-pages/SKILL.md @@ -214,6 +214,28 @@ column colStatus (attribute: "Status") { dropdownfilter f4 } -- Enumeration -- Boolean columns: omit the filter entirely ``` +**The filter goes inside the column's own braces.** A `filter { … }` block is +the GALLERY spelling of a different thing — the widget-wide filter bar, which a +data grid calls `controlbar`: + +```sql +-- ✅ data grid: per-column filter, inside the column +datagrid dg (...) { column colName (attribute: Name) { textfilter f1 } } + +-- ✅ gallery: the widget-wide filter bar, which the gallery calls `filter` +gallery g (...) { filter f { textfilter f1 } } + +-- ❌ the gallery form on a data grid — MDL-WIDGET29 +datagrid dg (...) { column colName (attribute: Name) filter f { textfilter f1 } } +``` + +That last line is worth reading twice: it is not a column with a filter block. +A widget is `type name (props) { body }`, so with the `filter` *outside* the +column's braces it parses as a column with **no body** followed by a separate +`filter` widget — which the grid has nowhere to put. It used to be dropped on +write with no diagnostic, so `DESCRIBE PAGE` showing a filterless column was the +only symptom; it is now refused at check and exec time. + ## NewEdit Page Template Form for creating or editing a single entity. **Requires a page parameter** to receive the object. diff --git a/.claude/skills/mendix/write-oql-queries/SKILL.md b/.claude/skills/mendix/write-oql-queries/SKILL.md index 074d0dd3e..09c9d1633 100644 --- a/.claude/skills/mendix/write-oql-queries/SKILL.md +++ b/.claude/skills/mendix/write-oql-queries/SKILL.md @@ -349,6 +349,31 @@ create view entity Module.ViewName ( ); ``` +### Step 1b: Know the two clause orders + +Mendix OQL accepts the select list in either position, and mxcli reads both: + +```sql +-- Select-first. Write new views this way; the rest of this skill assumes it. +select c.Name as Name, count(o.ID) as Orders +from Shop.Customer as c +group by c.Name + +-- From-first. Same query. This is what STUDIO PRO STORES, so it is what +-- `describe entity` gives you back — copy it, edit it, exec it unchanged. +from Shop.Customer as c +group by c.Name +select c.Name as Name, count(o.ID) as Orders +``` + +Note where `group by` sits: in the from-first order every clause except +`order by` / `limit` comes **before** the select list, and the grammar enforces +that. `from … select … group by …` is a parse error, not a variant. + +Do not rewrite a described view into select-first just to make it look +familiar — the stored text is what MxBuild validates against, and a needless +rewrite is a diff for nothing. + ### Step 2: Write SELECT Clause - Use **lowercase** aggregate functions: `sum()`, `avg()`, `count()` - Use `count(entity.ID)` not `count(*)` diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 32d026b23..28f2ddd0b 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -124,7 +124,16 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { }, Syntax: "-- Containers\nLAYOUTGRID name { ROW r { COLUMN c (DesktopWidth: 6) { ... } } }\nCONTAINER name (Class: 'cls') { ... }\nCONTAINER name (OnClick: MICROFLOW Module.MF) { ... } -- clickable container\nCUSTOMCONTAINER name (Class: 'cls') { ... }\nGROUPBOX name (Caption: 'C') { ... }\nTABCONTAINER name { TABPAGE tp (Caption: 'One') { ... } TABPAGE tp2 (Caption: 'Two') { ... } }\n\n" + "-- Data widgets\nDATAVIEW name (DataSource: $Param) { ... FOOTER f { ... } }\nDATAGRID name (DataSource: DATABASE Module.Entity) { COLUMN c (Attribute: A) }\nGALLERY name (DataSource: DATABASE Module.Entity, DesktopColumns: 3) { ... }\nLISTVIEW name (DataSource: DATABASE Module.Entity) { ... }\nLISTVIEW name (...) { ... TEMPLATE FOR Module.Specialization { ... } }\n\n" + - "-- Data grid filters and sort (inside a DATAGRID's FILTER block)\nDATAGRID dg (...) { COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (Attribute: A) } }\nTEXTFILTER | NUMBERFILTER | DATEFILTER | DROPDOWNFILTER | DROPDOWNSORT\n\n" + + "-- Data grid 2 column filters go INSIDE the column's own braces\nDATAGRID dg (...) { COLUMN c (Attribute: A) { TEXTFILTER tf (Attribute: A) } }\nTEXTFILTER | NUMBERFILTER | DATEFILTER | DROPDOWNFILTER | DROPDOWNSORT\n" + + "-- Match the filter to the column's type, or MxBuild refuses it: String ->\n" + + "-- TEXTFILTER, Integer/Long/Decimal -> NUMBERFILTER, Date and time -> DATEFILTER,\n" + + "-- Enumeration -> DROPDOWNFILTER. A Boolean column takes no filter at all.\n" + + "-- The grid-wide filter bar is CONTROLBAR; a GALLERY spells that same slot\n" + + "-- FILTER, so `FILTER f { ... }` belongs to a gallery and not to a datagrid:\n" + + "GALLERY g (...) { FILTER f { TEXTFILTER tf (Attribute: A) } }\n" + + "-- A FILTER block written on a DATAGRID is not a column filter and not a\n" + + "-- container the grid declares — it used to be dropped on write with no\n" + + "-- diagnostic, and is now refused (MDL-WIDGET29).\n\n" + "-- Inputs\nTEXTBOX name (Label: 'L', Attribute: Attr)\nTEXTAREA | DATEPICKER | COMBOBOX | CHECKBOX | RADIOBUTTONS\n\n" + "-- Actions\nACTIONBUTTON name (Caption: 'C', Action: SAVE_CHANGES, ButtonStyle: Primary)\nLINKBUTTON name (Caption: 'C', Action: ...)\n\n" + "-- Display\nDYNAMICTEXT name (Content: 'Hello, {1}!', ContentParams: [{1} = Name])\nTITLE name (Content: 'Heading')\nIMAGE name (Image: 'Module.Collection.ImageName')\nIMAGE name (ImageType: imageUrl, ImageUrl: 'https://…')\n" + diff --git a/docs-wiki/bug-patterns/silent-property-drop.md b/docs-wiki/bug-patterns/silent-property-drop.md index c173f19b7..4498fcc92 100644 --- a/docs-wiki/bug-patterns/silent-property-drop.md +++ b/docs-wiki/bug-patterns/silent-property-drop.md @@ -83,6 +83,29 @@ writers → both readers → describe. that `check` rejected outright; a validator that no pass calls reports no violations and reads exactly like a clean project. +**The sharper version: a check can be switched off by its own input.** Where a +family of rules hangs off one parsed fragment — a select clause, a resolved +definition, a decoded sub-document — a parse that yields nothing does not report +"I could not read this"; it reports nothing at all, which is the same output as +a clean file. The view-entity OQL scanner did this twice, two months apart and +in the same function: once because it compared cases wrongly, once because +Mendix's other clause order put the terminator behind the keyword it scanned +forward from. Both times a single visible false positive was the only hint, +while three real rules stopped running behind it. The durable fix is not the +next input shape, it is making *unreadable* a reported outcome distinct from +*nothing to report* — after which a third shape costs a diagnostic rather than a +blind spot. + +**Children drop the same way properties do.** A widget's body is distributed by +several passes that each skip what they do not recognise, so a child matching no +container, no slot and no catch-all is built and discarded exactly as an +unrouted property is. The compounding factor is that the *same slot* wears +different keywords on different widgets — the filters placeholder is `filter` on +a Gallery and `controlbar` on a Data Grid — so a form copied between two widgets +is both plausible and inert. That table is also the fix: because the engine +already maps keyword to property per widget, the diagnostic can name the +spelling *this* widget uses instead of listing everything it declares. + ## See also - [fix-issue findings](../../.claude/skills/fix-issue/findings/) — the individual diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 63927757d..f6143404c 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -62,6 +62,7 @@ create persistent entity Module.Photo ( | Create with extends | `create persistent entity Module.Name extends Parent.Entity (attrs);` | EXTENDS before `(` | | Create with auditing | `create persistent entity Module.Name (attrs, owner: autoowner, ChangedBy: autochangedby, CreatedDate: autocreateddate, ChangedDate: autochangeddate);` | Pseudo-types like AutoNumber | | Create view entity | `create view entity Module.Name (attrs) as select ...;` | OQL-backed read-only | +| View entity clause order | `... as select … from …;` **or** `... as from … group by … select …;` | Both are Mendix OQL and both are checked. The second is what **Studio Pro stores**, so it is what `DESCRIBE ENTITY` emits — describe → edit → exec round-trips. The declared attributes are matched to the select columns **by position**, in either order | | Create external entity | `create external entity Module.Name from odata client Module.Client (...) (attrs);` | From consumed OData | | Create external entities | `create [or modify] external entities from Module.Client [into module] [entities (...)];` | Bulk from $metadata | | Drop entity | `drop entity Module.Name;` | | @@ -1338,6 +1339,8 @@ MDL uses explicit property declarations for pages: | Page CSS class / style | `Class: 'css-class', Style: 'css: rule'` | `(Title: 'Home', Class: 'container-fluid bg-light', Style: 'min-height: 100vh')` — the page's Appearance | | Page variables | `variables: { $name: type = 'expr' }` | `variables: { $show: boolean = 'true' }` | | Repeated widget entries | ` ( … )` **in the widget body** | A repeatable property (FileUploader `allowedFileFormats`, HTML Element `attributes`, a chart's `series`) is a block, never a property value. `attributes: [(attributeName: 'x')]` is **MDL-WIDGET27** — it used to check clean, exec, and vanish from storage. `describe widget -p app.mpr` lists the container keywords | +| Data grid 2 column filter | `column c (attribute: A) { textfilter f }` | **Inside the column's braces.** `column c (…) filter f { … }` is the GALLERY form — the grammar reads it as a column with no body plus a sibling `filter` widget, which the grid has nowhere to put; it used to be dropped on write and is now **MDL-WIDGET29**. A grid-wide filter bar is `controlbar`; a gallery spells that same slot `filter`. Match the filter to the column's type (String → `textfilter`, number → `numberfilter`, DateTime → `datefilter`, Enumeration → `dropdownfilter`, Boolean → none) | +| Widget with nowhere to go | any widget in a pluggable widget's body | A child matching no container, slot or `template` catch-all is **MDL-WIDGET29** at check time and refused by `exec`. `describe widget -p app.mpr` lists what the parent declares. Needs the parent's definition, so it is silent without `-p` | | Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | diff --git a/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl b/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl new file mode 100644 index 000000000..12a47c523 --- /dev/null +++ b/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl @@ -0,0 +1,96 @@ +-- Bug: a widget written in a pluggable widget's body that the widget has +-- nowhere to put was BUILT AND THROWN AWAY, with no diagnostic from `check` +-- and no error from `exec`. `DESCRIBE PAGE` afterwards showing the widget gone +-- was the only way to find out. +-- +-- Reported by ako/view-entity-examples FINDINGS §2, as a Data Grid 2 filter. +-- The form that vanished is the one `mxcli syntax page widgets` documented: +-- +-- DATAGRID dg (...) { COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (...) } } +-- +-- Two things are wrong with it, and only together do they hide: +-- 1. `FILTER` is the GALLERY keyword for the filters placeholder. DataGrid 2 +-- spells the same property `controlbar` (widgetSlotKeywordOverrides). +-- 2. `widgetV3` is `type name props? body?`, so `COLUMN c (…) FILTER f { … }` +-- is a COLUMN WITH NO BODY followed by a SIBLING filter widget — nothing +-- binds the second to the first. The sibling landed among the grid's own +-- children, matched no container, and was dropped. +-- +-- Root cause: the four passes that place a pluggable widget's children +-- (applyChildSlots, applyObjectLists, and two auto-discovery passes) each +-- `continue` past a child they do not recognise, and applyChildSlots parks +-- the leftovers in `defaultWidgets` — which is assigned only if the widget +-- declares a `template` slot. A Gallery has one; DataGrid 2 does not, so its +-- leftovers were discarded. Not datagrid-specific: one `continue` per pass, +-- shared by every pluggable widget. +-- +-- Fix: MDL-WIDGET29 at check time and a refusal in buildPluggable at exec time, +-- both from unroutedPluggableChildren() so they cannot disagree. The message +-- names the spelling THIS widget uses for the same slot, since an author who +-- copied a gallery example needs `controlbar`, not a list to search. +-- +-- This repro is a plain .mdl, not .fail.mdl, on purpose: MDL-WIDGET29 needs the +-- parent widget's DEFINITION, which comes from the project's installed .mpk, and +-- `make check-mdl` runs `check` with no project (see the Makefile's note on +-- #891/#892). The refusal is covered by widget_unrouted_children_test.go; this +-- file records the working spellings so they stay working. +-- +-- Reading the definition from the project's .mpk is also what keeps the rule +-- current: Data Widgets ships DataGrid 2 in every Mendix 11 app, and upgrading +-- that module from the Marketplace changes what mxcli sees with no mxcli release. +-- +-- Measured end to end on Mendix 11.13.0, DataGrid 2 (Data Widgets) 3.4.0, with +-- the guard stubbed as the control: +-- guard stubbed, `exec --no-check` the FILTER form +-- → "Created page", exit 0 … and `describe page` shows +-- `column MeterCode (Attribute: MeterCode, Caption: 'Meter')` with NO +-- filter. That is the reported defect: a success that lost the widget. +-- guard in place, same script → refused, exit 1, naming `controlbar` +-- guard in place, THIS file → "Replaced page", and `describe page` shows +-- `column … { textfilter tf }` — it persisted +-- mx check on the result → 0 errors + +create module VFilterDrop; + +create entity VFilterDrop.Meter ( + MeterCode: String(50), + Readings: Integer, + LastSeen: DateTime +); + +create page VFilterDrop.Meters ( + title: 'Meters', + layout: Atlas_Core.Atlas_Default +) { + datagrid dgMeters (datasource: database VFilterDrop.Meter) { + -- The working form: the filter goes INSIDE the column's own braces, and + -- must match the column attribute's type. + column colCode (attribute: MeterCode, caption: 'Meter') { + textfilter fCode (attribute: MeterCode) + } + column colReadings (attribute: Readings, caption: 'Readings') { + numberfilter fReadings (attribute: Readings) + } + column colSeen (attribute: LastSeen, caption: 'Last seen') { + datefilter fSeen (attribute: LastSeen) + } + } +} + +-- Control: on a GALLERY the very same `filter { … }` block is correct — it is +-- that widget's own keyword for the filters placeholder. A rule that simply +-- banned the keyword would have broken this page, which is why the fix asks +-- the parent's definition rather than matching on the word. +create page VFilterDrop.MeterGallery ( + title: 'Meter gallery', + layout: Atlas_Core.Atlas_Default +) { + gallery galMeters (datasource: database VFilterDrop.Meter, desktopcolumns: 3) { + filter galFilter { + textfilter fGalCode (attribute: MeterCode) + } + template galTemplate { + dynamictext txtCode (content: '{1}', contentparams: [{1} = MeterCode]) + } + } +} diff --git a/mdl-examples/bug-tests/view-entity-oql-from-first-alias.fail.mdl b/mdl-examples/bug-tests/view-entity-oql-from-first-alias.fail.mdl new file mode 100644 index 000000000..5ba35fb05 --- /dev/null +++ b/mdl-examples/bug-tests/view-entity-oql-from-first-alias.fail.mdl @@ -0,0 +1,31 @@ +-- Negative half of view-entity-oql-from-first-clause-order.mdl. +-- +-- The reported symptom of the from-first blind spot was one bogus error. The +-- unreported half was worse: with the select clause unreadable, every column +-- rule stopped running, so a from-first query got NO checking at all. +-- +-- This file is that proof. A select column with no `as` alias is MDL030's +-- reason for existing — MxBuild rejects the view (CE0174) — and in the +-- from-first order it used to pass `check` in silence. +-- +-- Expected: `mxcli check` FAILS with MDL030 "select column 1 has no as alias". +-- Before the fix: exit 0, no diagnostic. +-- +-- The working forms are in the sibling file. + +create module VClauseOrderFail; + +create entity VClauseOrderFail.Reading ( + MeterCode: String(50), + Kwh: Decimal +); + +create view entity VClauseOrderFail.MeterTotals ( + MeterCode: String(50), + Readings: Integer +) as ( + from VClauseOrderFail.Reading as r + group by r.MeterCode + select r.MeterCode, + count(r.Kwh) as Readings +); diff --git a/mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl b/mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl new file mode 100644 index 000000000..913f86671 --- /dev/null +++ b/mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl @@ -0,0 +1,87 @@ +-- Bug: Mendix OQL has two clause orders, and mxcli's view-entity checker could +-- read only one of them. +-- +-- select … from … group by … -- what the skills teach; worked +-- from … group by … select … -- what STUDIO PRO STORES; did not +-- +-- The second is not an exotic spelling. It is the canonical Mendix form, so it +-- is what a `DESCRIBE ENTITY` of any Studio-Pro-authored view hands back — +-- which made describe → edit → check → exec, the obvious loop, the one loop +-- that broke. Feeding a describe straight back reported: +-- +-- statement 1: view entity 'X' has type mismatches: +-- - could not parse select clause from OQL query +-- +-- Reported by ako/view-entity-examples FINDINGS §3. +-- +-- Root cause: `extractSelectClause` found SELECT and then scanned FORWARD for +-- FROM to end the column list. In the from-first order the FROM is BEHIND the +-- SELECT, so the scan ran off the end and returned "". That empty string did +-- two things, and only the first was visible: +-- 1. inferOQLTypes reported "could not parse select clause" — one bogus error; +-- 2. every column rule sits behind `if selectClause != ""`, so MDL030 +-- (missing alias) and MDL072 (quoted alias) SILENTLY STOPPED RUNNING. +-- The grammar had accepted both orders all along (`oqlQueryTerm` in +-- MDLCatalog.g4) — only the hand-written scanner had not caught up. +-- +-- This is a recurrence at the same function: bug 9b was a case-comparison +-- slip that made it return "" for EVERY query, with the same two symptoms. +-- An unreadable select clause is now REPORTED rather than skipped, so a +-- third clause shape cannot switch the checker off in silence again. +-- +-- Verify: +-- ./bin/mxcli check mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl +-- → clean (before the fix: "could not parse select clause" with a project) +-- Its sibling proves the rules really run on this order: +-- ./bin/mxcli check mdl-examples/bug-tests/view-entity-oql-from-first-alias.fail.mdl +-- → MDL030 on the column with no alias (before the fix: silence) + +create module VClauseOrder; + +create entity VClauseOrder.Reading ( + MeterCode: String(50), + Kwh: Decimal +); + +-- The from-first order, with every clause it can carry before the select list. +create view entity VClauseOrder.MeterTotals ( + MeterCode: String(50), + Readings: Integer, + TotalKwh: Decimal +) as ( + from VClauseOrder.Reading as r + where r.Kwh > 0 + group by r.MeterCode + select r.MeterCode as MeterCode, + count(r.Kwh) as Readings, + sum(r.Kwh) as TotalKwh +); + +-- Control: the same view in the select-first order. Both must check clean and +-- be read identically — if only one does, the blind spot has just moved. +create view entity VClauseOrder.MeterTotalsSelectFirst ( + MeterCode: String(50), + Readings: Integer, + TotalKwh: Decimal +) as ( + select r.MeterCode as MeterCode, + count(r.Kwh) as Readings, + sum(r.Kwh) as TotalKwh + from VClauseOrder.Reading as r + where r.Kwh > 0 + group by r.MeterCode +); + +-- Control: ORDER BY and LIMIT end a from-first select list, and they are also +-- ordinary names. Widening the terminator set has to not cut a real query in +-- half — `o.Order` here is an attribute, not a clause. +create view entity VClauseOrder.TopMeters ( + MeterCode: String(50), + TotalKwh: Decimal +) as ( + from VClauseOrder.Reading as r + group by r.MeterCode + select r.MeterCode as MeterCode, sum(r.Kwh) as TotalKwh + order by r.MeterCode asc + limit 10 +); diff --git a/mdl/executor/oql_type_inference.go b/mdl/executor/oql_type_inference.go index b90e57bc7..50db3e941 100644 --- a/mdl/executor/oql_type_inference.go +++ b/mdl/executor/oql_type_inference.go @@ -32,7 +32,8 @@ func inferOQLTypes(ctx *ExecContext, oqlQuery string, declaredAttrs []ast.ViewAt // Extract SELECT clause selectClause := extractSelectClause(oqlQuery) if selectClause == "" { - warnings = append(warnings, "could not parse select clause from OQL query") + warnings = append(warnings, + "could not parse select clause from OQL query, so no column was type-checked") return columns, warnings } @@ -451,65 +452,151 @@ func validateViewEntityTypes(ctx *ExecContext, stmt *ast.CreateViewEntityStmt) [ return errors } -// extractSelectClause extracts the SELECT clause from an OQL query. -// Handles subqueries by tracking parenthesis depth to find the main FROM clause. +// Mendix OQL has TWO clause orders and the MDL grammar accepts both +// (`oqlQueryTerm`, MDLCatalog.g4): +// +// SELECT … FROM … GROUP BY … -- select first; FROM ends the list +// FROM … GROUP BY … SELECT … -- Mendix's own canonical spelling +// +// The second is not exotic: it is what Studio Pro stores, so it is what +// `DESCRIBE ENTITY` hands back, and describe → check → exec goes through it +// every time. +// +// The two take DIFFERENT terminator sets, and deliberately so. In the +// select-first order FROM ends the list and ORDER/LIMIT cannot — they sit past +// the FROM — so admitting them there would cut `select o.Limit as Limit from …` +// in half and report the remaining columns against the wrong attributes. Only +// the from-first order needs the wider set, because there the list runs to the +// end of the query. +var ( + selectFirstTerminators = []string{"FROM", "UNION"} + fromFirstTerminators = []string{"UNION", "ORDER BY", "LIMIT", "OFFSET"} +) + +// extractSelectClause extracts the SELECT clause from an OQL query, in either +// clause order. Empty string means the query has no readable select list — +// callers MUST treat that as "could not read", never as "nothing to check": +// every column-level rule hangs off this one string, so a silent "" turns the +// whole view-entity checker off. ValidateOQLSyntax reports it for that reason. func extractSelectClause(oql string) string { + clause, _ := extractSelectClauseOK(oql) + return clause +} + +// extractSelectClauseOK is extractSelectClause plus whether a top-level SELECT +// was found at all. The two outcomes need separating because they call for +// different things: no SELECT is a query the grammar would already have +// rejected, while a SELECT whose list could not be read is a checker that has +// quietly stopped checking. +func extractSelectClauseOK(oql string) (string, bool) { // Normalize whitespace oql = strings.TrimSpace(oql) upperOql := strings.ToUpper(oql) - // Find SELECT keyword. Compare uppercase-to-uppercase: upperOql is already - // upper-cased, so the needle must be too (a lowercase needle never matches). - selectIdx := strings.Index(upperOql, "SELECT") - if selectIdx == -1 { - return "" + // Find the top-level SELECT. Depth- and quote-aware, so a subquery's SELECT + // in a from-first query (`from (select …) as t select …`) is not mistaken + // for the outer one. + selectIdx := topLevelKeywordIndex(oql, upperOql, 0, "SELECT") + if selectIdx < 0 { + return "", false + } + startIdx := selectIdx + len("SELECT") + + // Which clause order this is, is decided by what comes FIRST, not by what + // is present: a select-first query has a FROM too. + terminators := selectFirstTerminators + if fromIdx := topLevelKeywordIndex(oql, upperOql, 0, "FROM"); fromIdx >= 0 && fromIdx < selectIdx { + terminators = fromFirstTerminators } - // Start after SELECT keyword - startIdx := selectIdx + 6 // len("SELECT") + endIdx := topLevelKeywordIndex(oql, upperOql, startIdx, terminators...) + if endIdx < 0 { + // No terminator: the list runs to the end of the query. That is the + // ordinary shape of a from-first query (`from … select a as A`), and + // for a select-first one it means a FROM-less query, whose column list + // is likewise the rest — either way there is something to check, and + // reporting "unreadable" here would be a checker refusing its own input. + endIdx = len(oql) + } + return strings.TrimSpace(oql[startIdx:endIdx]), true +} - // Find the main FROM clause or UNION (not inside subqueries). Slice from - // upperOql (same byte offsets as oql for ASCII) so keyword comparisons are - // case-consistent — comparing strings.ToUpper(...) to a lowercase literal - // could never match and made this function always return "" (bug 9b). +// topLevelKeywordIndex returns the byte offset of the first of words appearing +// at parenthesis depth 0, on a word boundary, at or after start — or -1. +// +// upperOql must be strings.ToUpper(oql) and words must already be upper-case: +// the comparison is uppercase-to-uppercase, because a lowercase needle against +// an upper-cased haystack never matches and made this whole family return "" +// for every query once already (bug 9b). +// +// Quoted runs are skipped. OQL takes double-quoted identifiers exactly as SQL +// does, and mxcli passes them through so a reserved word survives MxBuild — +// so `select s."Order" as OrderValue` is a query a user really writes, and +// matching the ORDER inside those quotes would cut the select list in half. +// +// A word containing a space is a PHRASE, and the space matches any run of +// whitespace: "ORDER BY" must be spelled that way rather than as "ORDER", +// because a bare ORDER also occurs as an ordinary name. +func topLevelKeywordIndex(oql, upperOql string, start int, words ...string) int { depth := 0 - for i := startIdx; i < len(oql); i++ { - ch := oql[i] - switch ch { + for i := start; i < len(oql); i++ { + switch c := oql[i]; c { case '(': depth++ + continue case ')': depth-- - default: - if depth == 0 { - // Check for FROM keyword at depth 0 - if i+4 <= len(oql) { - word := upperOql[i : i+4] - if word == "FROM" { - // Make sure it's a word boundary (not part of another identifier) - prevOk := i == startIdx || !isIdentChar(oql[i-1]) - nextOk := i+4 >= len(oql) || !isIdentChar(oql[i+4]) - if prevOk && nextOk { - return strings.TrimSpace(oql[startIdx:i]) - } - } - } - // Check for UNION keyword at depth 0 (ends current query term) - if i+5 <= len(oql) { - word := upperOql[i : i+5] - if word == "UNION" { - prevOk := i == startIdx || !isIdentChar(oql[i-1]) - nextOk := i+5 >= len(oql) || !isIdentChar(oql[i+5]) - if prevOk && nextOk { - return strings.TrimSpace(oql[startIdx:i]) - } - } - } + continue + case '\'', '"', '`': + // Skip to the closing quote. A doubled quote (OQL's own escape, + // 'it''s') reads as two adjacent runs, which lands in the same place. + for i++; i < len(oql) && oql[i] != c; i++ { } + continue + } + if depth != 0 { + continue + } + for _, w := range words { + end, ok := matchPhraseAt(oql, upperOql, i, w) + if !ok { + continue + } + prevOk := i == 0 || !isIdentChar(oql[i-1]) + nextOk := end >= len(oql) || !isIdentChar(oql[end]) + if prevOk && nextOk { + return i + } + } + } + return -1 +} + +// matchPhraseAt reports whether phrase matches oql at offset i, and the offset +// just past the match. A space in phrase matches one or more whitespace +// characters; every other character is compared upper-case against upperOql. +func matchPhraseAt(oql, upperOql string, i int, phrase string) (int, bool) { + for p := 0; p < len(phrase); p++ { + if phrase[p] == ' ' { + start := i + for i < len(oql) && isOQLSpace(oql[i]) { + i++ + } + if i == start { + return 0, false + } + continue + } + if i >= len(oql) || upperOql[i] != phrase[p] { + return 0, false } + i++ } + return i, true +} - return "" +func isOQLSpace(ch byte) bool { + return ch == ' ' || ch == '\t' || ch == '\n' || ch == '\r' } // isIdentChar returns true if ch is a valid identifier character. @@ -1029,8 +1116,27 @@ func ValidateOQLSyntax(oql string) []linter.Violation { } } - // Check that all top-level SELECT columns have explicit AS aliases - selectClause := extractSelectClause(oql) + // Check that all top-level SELECT columns have explicit AS aliases. + // + // An unreadable select list is reported rather than skipped. Every rule + // below hangs off this one string, so returning "" used to switch the + // column checks off without saying so — which is how the FROM-first clause + // order went unnoticed: the only visible symptom was ONE bogus error from + // inferOQLTypes, while MDL030 and MDL072 quietly stopped running. A checker + // that cannot read its input has to say so. + selectClause, hasSelect := extractSelectClauseOK(oql) + if hasSelect && selectClause == "" { + violations = append(violations, linter.Violation{ + RuleID: "MDL030", + Severity: linter.SeverityError, + Message: "the OQL has a select clause but its column list could not be read, " + + "so no column was checked (alias, type and length rules all skipped)", + Location: linter.Location{DocumentType: "viewentity"}, + Suggestion: "Both clause orders are supported — `select … from …` and Mendix's own " + + "`from … group by … select …`. If the query is one of those and still " + + "lands here, it is an mxcli gap: please report it with the query.", + }) + } if selectClause != "" { columns := parseSelectColumns(selectClause) aliasPattern := oqlAliasSuffixRe diff --git a/mdl/executor/oql_type_inference_test.go b/mdl/executor/oql_type_inference_test.go index eb0491ca9..dc5518011 100644 --- a/mdl/executor/oql_type_inference_test.go +++ b/mdl/executor/oql_type_inference_test.go @@ -64,11 +64,14 @@ func TestExtractSelectClause(t *testing.T) { want: "a", }, { - // OQL always requires a FROM; a FROM-less query is malformed, so - // returning "" (→ "could not parse select clause") is acceptable here. - name: "no from clause returns empty", + // Was "": the scanner needed a FROM to know where the list ended, + // so a FROM-less query read as unreadable. The list is simply the + // rest of the query, and saying so matters now that an unreadable + // clause is REPORTED rather than skipped — "" here would refuse a + // query over a column the checker can perfectly well check. + name: "no from clause reads to the end of the query", oql: "select 1", - want: "", + want: "1", }, { name: "from inside subquery is not the main FROM", diff --git a/mdl/executor/validate_oql_clause_order_test.go b/mdl/executor/validate_oql_clause_order_test.go new file mode 100644 index 000000000..68cb06916 --- /dev/null +++ b/mdl/executor/validate_oql_clause_order_test.go @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// Mendix OQL has two clause orders and the MDL grammar accepts both +// (`oqlQueryTerm`, MDLCatalog.g4). Studio Pro stores the from-first one, so it +// is what `DESCRIBE ENTITY` emits — and extractSelectClause could only read the +// select-first one, so feeding a describe straight back to `check` reported +// `could not parse select clause from OQL query` on a query Mendix itself had +// written. +// +// Every test here pairs the from-first query with its select-first twin as the +// control: the two must be read and judged identically, or the fix is only +// moving the blind spot. + +// fromFirst and selectFirst are the same query in the two clause orders. +const ( + fromFirstOQL = `from Sales.Order as o group by o.Number select o.Number as Number, count(o.ID) as Lines` + selectFirstOQL = `select o.Number as Number, count(o.ID) as Lines from Sales.Order as o group by o.Number` +) + +func TestExtractSelectClause_ReadsBothClauseOrders(t *testing.T) { + const want = "o.Number as Number, count(o.ID) as Lines" + + if got := extractSelectClause(fromFirstOQL); got != want { + t.Errorf("from-first: extractSelectClause = %q, want %q", got, want) + } + // Control: the order that always worked must be unchanged. + if got := extractSelectClause(selectFirstOQL); got != want { + t.Errorf("select-first (control): extractSelectClause = %q, want %q", got, want) + } +} + +func TestExtractSelectClause_FromFirstTerminators(t *testing.T) { + const cols = "o.Number as Number" + cases := []struct { + name string + oql string + want string + }{ + { + // The ordinary shape: nothing follows the select list at all. + name: "runs to the end of the query", + oql: `from Sales.Order as o select o.Number as Number`, + want: cols, + }, + { + name: "order by ends the list", + oql: `from Sales.Order as o select o.Number as Number order by o.Number limit 10`, + want: cols, + }, + { + name: "limit ends the list", + oql: `from Sales.Order as o select o.Number as Number limit 10`, + want: cols, + }, + { + name: "union ends the first query term", + oql: `from Sales.Order as o select o.Number as Number union from Sales.Quote as q select q.Number as Number`, + want: cols, + }, + { + // A subquery's own SELECT must not be mistaken for the outer one, + // which in this order comes AFTER it. + name: "subquery select is not the outer select", + oql: `from (from Sales.Line as l select l.OrderID as OrderID) as t select t.OrderID as Number`, + want: "t.OrderID as Number", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := extractSelectClause(tc.oql); got != tc.want { + t.Errorf("extractSelectClause(%q) = %q, want %q", tc.oql, got, tc.want) + } + }) + } +} + +// TestExtractSelectClause_OrderAndLimitAreNamesToo is the control for widening +// the terminator set. ORDER/LIMIT/OFFSET end a from-first select list, and they +// are also perfectly ordinary attribute and alias names — so admitting them in +// the SELECT-FIRST order, where they cannot legally appear before the FROM, +// would cut a real query's list in half and then report every remaining column +// against the wrong declared attribute. +func TestExtractSelectClause_OrderAndLimitAreNamesToo(t *testing.T) { + cases := []struct { + name, oql, want string + }{ + { + name: "select-first: a column named Limit is not a LIMIT clause", + oql: `select o.Limit as Limit, o.Number as Number from Sales.Order as o`, + want: "o.Limit as Limit, o.Number as Number", + }, + { + name: "select-first: a column named Order is not an ORDER BY", + oql: `select o.Order as Order, o.Number as Number from Sales.Order as o`, + want: "o.Order as Order, o.Number as Number", + }, + { + // Even in from-first, a bare ORDER is a name: only the PHRASE + // "order by" ends the list. + name: "from-first: a bare Order alias is not an ORDER BY", + oql: `from Sales.Order as o select o.Number as Order`, + want: "o.Number as Order", + }, + { + // A quoted identifier is how a reserved word survives MxBuild, and + // mxcli passes the quotes through — so the scanner has to skip + // quoted runs or it matches the keyword inside one. + name: "from-first: a quoted reserved word in a source position", + oql: `from Sales.Order as o select o."Order" as OrderValue, o.Number as Number`, + want: `o."Order" as OrderValue, o.Number as Number`, + }, + { + name: "from-first: a string literal containing a keyword", + oql: `from Sales.Order as o select 'order by' as Label, o.Number as Number`, + want: `'order by' as Label, o.Number as Number`, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := extractSelectClause(tc.oql); got != tc.want { + t.Errorf("extractSelectClause(%q) = %q, want %q", tc.oql, got, tc.want) + } + }) + } +} + +// TestFromFirstOQL_IsCheckedNotSkipped is the point of the fix. The reported +// symptom was one bogus error; the unreported half was that MDL030 and MDL072 +// stopped running entirely, because both sit behind `if selectClause != ""`. +// A checker that goes quiet is worse than one that complains. +func TestFromFirstOQL_IsCheckedNotSkipped(t *testing.T) { + // A column with no alias at all — MDL030's reason for existing. + const missingAlias = `from Sales.Order as o group by o.Number select o.Number, count(o.ID) as Lines` + + ids := oqlRuleIDs(missingAlias) + if !hasOQLRule(ids, "MDL030") { + t.Errorf("from-first: MDL030 did not fire on a column with no alias, got %v", ids) + } + // Control: the same defect in the order that always worked. + if ids := oqlRuleIDs(`select o.Number, count(o.ID) as Lines from Sales.Order as o`); !hasOQLRule(ids, "MDL030") { + t.Errorf("select-first (control): MDL030 did not fire, got %v", ids) + } + // Control: a well-formed from-first query is not newly complained about. + if ids := oqlRuleIDs(fromFirstOQL); len(ids) != 0 { + t.Errorf("from-first well-formed query now reports %v, want none", ids) + } +} + +func TestFromFirstOQL_NoLongerReportsCouldNotParse(t *testing.T) { + attrs := []ast.ViewAttribute{ + {Name: "Number", Type: ast.DataType{Kind: ast.TypeString, Length: 200}}, + {Name: "Lines", Type: ast.DataType{Kind: ast.TypeInteger}}, + } + if v := ValidateOQLTypes(fromFirstOQL, attrs); len(v) != 0 { + t.Errorf("from-first: unexpected type violations %v", v) + } + + // And the type rules really do run now, rather than passing vacuously: + // count() is Integer, so declaring it Decimal must be caught in BOTH orders. + wrong := []ast.ViewAttribute{ + {Name: "Number", Type: ast.DataType{Kind: ast.TypeString, Length: 200}}, + {Name: "Lines", Type: ast.DataType{Kind: ast.TypeDecimal}}, + } + for _, c := range []struct{ name, oql string }{ + {"from-first", fromFirstOQL}, + {"select-first (control)", selectFirstOQL}, + } { + v := ValidateOQLTypes(c.oql, wrong) + if len(v) != 1 || !strings.Contains(v[0].Message, "declared as Decimal") { + t.Errorf("%s: want one MDL031 about Decimal, got %v", c.name, v) + } + } +} + +// TestUnreadableSelectClauseIsReported pins the backstop. The FROM-first gap +// was invisible for as long as it was because an unreadable select clause +// turned the column rules off silently — so if a third clause shape ever +// arrives, the checker has to say it could not read the query rather than +// report nothing and look clean. +func TestUnreadableSelectClauseIsReported(t *testing.T) { + ids := oqlRuleIDs(`select from Sales.Order as o`) + if !hasOQLRule(ids, "MDL030") { + t.Fatalf("an empty select list was accepted in silence, got %v", ids) + } + found := false + for _, v := range ValidateOQLSyntax(`select from Sales.Order as o`) { + if strings.Contains(v.Message, "could not be read") { + found = true + } + } + if !found { + t.Error("the diagnostic does not say the columns went unchecked") + } + // Control: a query whose columns ARE readable must not collect it. + for _, oql := range []string{fromFirstOQL, selectFirstOQL} { + for _, v := range ValidateOQLSyntax(oql) { + if strings.Contains(v.Message, "could not be read") { + t.Errorf("readable query %q reported as unreadable", oql) + } + } + } +} + +// TestDescribeOutputOrderIsWhatMendixStores documents why the from-first order +// is the one that matters, so nobody later "simplifies" the terminator sets +// back to one. DESCRIBE ENTITY emits the stored OqlQuery verbatim +// (cmd_entities_describe.go), so describe → check is exactly this path. +func TestDescribeOutputOrderIsWhatMendixStores(t *testing.T) { + if strings.Contains(strings.ToUpper(fromFirstOQL), "SELECT") && + strings.Index(strings.ToUpper(fromFirstOQL), "FROM") > strings.Index(strings.ToUpper(fromFirstOQL), "SELECT") { + t.Fatal("fromFirstOQL is not actually from-first; the test fixture drifted") + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 72e5fbd05..240b643d4 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -156,6 +156,12 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // widgets get the stricter def.json check (MDL-WIDGET01) above, and // object-list items are validated by the object-list engine. def := lookupWidgetDef(w, registry) + // #2 from the view-entity-examples findings: a child the parent has + // nowhere to put. MDL-WIDGET26 above covers a container KEYWORD in that + // position; this covers a real widget, which resolves fine on its own and + // so gets past every other rule. Needs the parent's definition, and stays + // quiet without one for the same reason MDL-WIDGET26 does. + out = append(out, validateUnroutedChildren(w, def, locationPrefix)...) // A generic widget type that resolved to nothing is already reported as // MDL-WIDGET25 (the kind is wrong). Validating its properties on top of // that says the kind is fine and the property is not, which points at diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index bd478d3e7..546de4995 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -396,6 +396,14 @@ func (e *PluggableWidgetEngine) Build(def *WidgetDefinition, w *ast.WidgetV3) (* } } + // 4.0 Refuse a child none of the passes below can place. Each of them skips + // what it does not recognise, so without this the child is built and thrown + // away — a page that writes, builds and renders without the widget the + // author put there. See widget_unrouted_children.go. + if err := refuseUnroutedChildren(def, w); err != nil { + return nil, err + } + // 4.1 Apply child slots (.def.json) — skip children whose keyword belongs // to an objectLists mapping (handled by applyObjectLists below). objectListContainers := make(map[string]bool, len(def.ObjectLists)) diff --git a/mdl/executor/widget_unrouted_children.go b/mdl/executor/widget_unrouted_children.go new file mode 100644 index 000000000..49e4a2bac --- /dev/null +++ b/mdl/executor/widget_unrouted_children.go @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// A pluggable widget's body is distributed over four passes in buildPluggable — +// declared child slots, object lists, and two auto-discovery passes — and every +// one of them SKIPS a child it does not recognise. Nothing then reported the +// skip, so a child that matched none of the four was built and thrown away: +// +// datagrid dg (...) { +// column c (attribute: Name) filter f { textfilter tf (attribute: Name) } +// } +// +// `filter` is a Gallery keyword (widgetSlotKeywordOverrides spells the same +// DataGrid property `controlbar`), and the grammar reads that line as a COLUMN +// with no body followed by a SIBLING filter widget — so it landed among the +// grid's own children, matched nothing, and vanished. `mxcli check` was silent, +// `exec` succeeded, and `DESCRIBE PAGE` showing a column with no filter was the +// only way to find out. That spelling is what `mxcli syntax page widgets` +// documented, so it was also the likeliest thing for an author to write. +// +// The drop is not datagrid-specific — it is one `continue` per pass, shared by +// every pluggable widget — which is why this is a rule about children and slots +// rather than a special case for filters. + +// unroutedPluggableChildren returns the direct children of w that the engine +// has nowhere to put, given the parent's definition. +// +// It is deliberately CONSERVATIVE: it reports only what it can prove from the +// definition alone, so that `check` (which has the definition) never claims a +// drop that `exec` (which has the definition AND the widget template) would not +// make. The bail-outs below are each a case where a child may still be routed +// by information this function cannot see. +func unroutedPluggableChildren(def *WidgetDefinition, w *ast.WidgetV3) []*ast.WidgetV3 { + if def == nil || w == nil || len(w.Children) == 0 { + return nil + } + // No declared child slots: applyChildSlots returns early and the auto pass + // hands every unmatched child to the first widgets-typed property it finds. + // Something may well take them, and this function cannot see what. + if len(def.ChildSlots) == 0 { + return nil + } + // A `template` slot IS the catch-all — applyChildSlots assigns leftover + // children to it (defaultSlotContainer). A Gallery has one, which is why an + // arbitrary widget in a gallery body is fine and the same widget in a + // datagrid body is not. + if defHasDefaultChildSlot(def) { + return nil + } + var out []*ast.WidgetV3 + for _, c := range w.Children { + if c == nil || defRoutesChild(def, c) { + continue + } + out = append(out, c) + } + return out +} + +func defHasDefaultChildSlot(def *WidgetDefinition) bool { + for _, cs := range def.ChildSlots { + if strings.EqualFold(cs.MDLContainer, defaultSlotContainer) { + return true + } + } + return false +} + +// defRoutesChild mirrors the matching done by applyObjectLists, applyChildSlots +// and the auto-discovery passes. Matching is case-insensitive here even though +// the registry lower-cases MDLContainer on load, so that a hand-written +// .def.json cannot turn a casing slip into a false "this is dropped". +func defRoutesChild(def *WidgetDefinition, c *ast.WidgetV3) bool { + for _, ol := range def.ObjectLists { + // By container keyword (`column c { … }`), or by name against the + // property key (the auto-discovery pass matches a child's NAME). + if strings.EqualFold(ol.MDLContainer, c.Type) || strings.EqualFold(ol.PropertyKey, c.Name) { + return true + } + } + isContainer := strings.EqualFold(c.Type, "container") + for _, cs := range def.ChildSlots { + if strings.EqualFold(cs.MDLContainer, c.Type) || strings.EqualFold(cs.PropertyKey, c.Name) { + return true + } + // `container { … }` — routed by the container's NAME, the + // authoring form applyChildSlots supports for slots whose keyword reads + // badly as a widget type. + if isContainer && strings.EqualFold(cs.MDLContainer, c.Name) { + return true + } + } + return false +} + +// unroutedChildMessage explains one dropped child, naming what the parent does +// declare. The "spelled X on this widget" half is the answer in the case that +// prompted the rule: `filter` and `controlbar` are the SAME property under two +// widgets' conventions, so an author copying a gallery example onto a datagrid +// needs the other spelling, not a list to hunt through. +func unroutedChildMessage(def *WidgetDefinition, child *ast.WidgetV3) string { + msg := fmt.Sprintf("`%s` is not a container or slot of %s, so it would be dropped on write", + strings.ToLower(child.Type), parentLabel(def)) + if alt := sameSlotUnderAnotherName(def, child.Type); alt != "" { + msg += fmt.Sprintf(" — on this widget that slot is spelled `%s`", alt) + } + return msg + declaredContainers(def) +} + +// sameSlotUnderAnotherName finds the keyword THIS widget uses for the property +// that some other widget spells `keyword`. Derived from +// widgetSlotKeywordOverrides, which is where the convention already lives, so +// the two cannot drift. +func sameSlotUnderAnotherName(def *WidgetDefinition, keyword string) string { + var propertyKey string + for widgetID, slots := range widgetSlotKeywordOverrides { + if widgetID == def.WidgetID { + continue + } + for key, kw := range slots { + if strings.EqualFold(kw, keyword) { + propertyKey = key + } + } + } + if propertyKey == "" { + return "" + } + for _, cs := range def.ChildSlots { + if strings.EqualFold(cs.PropertyKey, propertyKey) { + return strings.ToLower(cs.MDLContainer) + } + } + return "" +} + +// validateUnroutedChildren is the check-time half (MDL-WIDGET29). It sits beside +// MDL-WIDGET26, which covers the neighbouring case: a container KEYWORD (`group`, +// `series` — words that are not widgets at all) under a parent that does not +// declare it. This one covers a real WIDGET in the same position, which +// MDL-WIDGET26 cannot report because a widget resolves perfectly well on its own. +func validateUnroutedChildren(w *ast.WidgetV3, def *WidgetDefinition, locationPrefix string) []linter.Violation { + var out []linter.Violation + for _, child := range unroutedPluggableChildren(def, w) { + out = append(out, linter.Violation{ + RuleID: "MDL-WIDGET29", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: %s", locationPrefix, unroutedChildMessage(def, child)), + Suggestion: "move it into one of the parent's containers, or out of the widget's body", + }) + } + return out +} + +// refuseUnroutedChildren is the exec-time half. `check` can be skipped +// (`exec --no-check`), and a write that silently discards part of what it was +// given is the failure this whole rule exists to stop — so the writer refuses +// rather than trusting that the checker ran. +func refuseUnroutedChildren(def *WidgetDefinition, w *ast.WidgetV3) error { + dropped := unroutedPluggableChildren(def, w) + if len(dropped) == 0 { + return nil + } + return mdlerrors.NewValidationf("widget `%s`: %s", w.Name, unroutedChildMessage(def, dropped[0])) +} diff --git a/mdl/executor/widget_unrouted_children_test.go b/mdl/executor/widget_unrouted_children_test.go new file mode 100644 index 000000000..a5ac4a45b --- /dev/null +++ b/mdl/executor/widget_unrouted_children_test.go @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// datagridLikeDef is DataGrid2's shape: a `columns` object list, two +// widgets-typed slots, and — the part that matters — NO `template` slot, so +// there is no catch-all for a child that matches nothing. +// +// The container spellings come from mdlContainerForWidgetSlot rather than being +// retyped, so this fixture cannot disagree with the convention table the engine +// and the diagnostic both read. +func datagridLikeDef() *WidgetDefinition { + const id = "com.mendix.widget.web.datagrid.Datagrid" + slot := func(key string) ChildSlotMapping { + return ChildSlotMapping{ + PropertyKey: key, + // The registry lower-cases this on load; do the same here. + MDLContainer: strings.ToLower(mdlContainerForWidgetSlot(id, key)), + Operation: "widgets", + } + } + return &WidgetDefinition{ + WidgetID: id, + MDLName: "DATAGRID", + WidgetKind: "pluggable", + ObjectLists: []ObjectListMapping{{PropertyKey: "columns", MDLContainer: "column"}}, + ChildSlots: []ChildSlotMapping{slot("emptyPlaceholder"), slot("filtersPlaceholder")}, + } +} + +func wdg(typ, name string, children ...*ast.WidgetV3) *ast.WidgetV3 { + return &ast.WidgetV3{Type: typ, Name: name, Children: children} +} + +func droppedTypes(def *WidgetDefinition, w *ast.WidgetV3) []string { + var out []string + for _, c := range unroutedPluggableChildren(def, w) { + out = append(out, c.Type) + } + return out +} + +// TestDataGridFilterBlockIsReportedNotSwallowed is the reported case. The MDL +// that `mxcli syntax page widgets` documented — +// +// COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (Attribute: A) } +// +// parses as a COLUMN with no body plus a SIBLING filter widget, because +// widgetV3 is `type name props? body?` and nothing binds the second widget to +// the first. That sibling used to be built and discarded in silence. +func TestDataGridFilterBlockIsReportedNotSwallowed(t *testing.T) { + grid := wdg("datagrid", "dg", + wdg("column", "colMeter"), + wdg("filter", "f", wdg("textfilter", "tf")), + ) + got := droppedTypes(datagridLikeDef(), grid) + if len(got) != 1 || got[0] != "filter" { + t.Fatalf("dropped children = %v, want exactly [filter]", got) + } +} + +// TestDataGridRoutedChildrenAreNotReported is the control. Everything the grid +// really does declare has to stay silent, or the rule just trades a silent drop +// for a wall of false refusals. +func TestDataGridRoutedChildrenAreNotReported(t *testing.T) { + def := datagridLikeDef() + for _, c := range []*ast.WidgetV3{ + wdg("column", "colMeter"), // the object list + wdg("controlbar", "cb"), // filtersPlaceholder, DataGrid's spelling + wdg("emptyplaceholder", "ep"), // the other widgets slot + wdg("container", "filtersPlaceholder"), // `container ` form + {Type: "container", Name: "emptyPlaceholder"}, // ditto, by property key + } { + if got := droppedTypes(def, wdg("datagrid", "dg", c)); got != nil { + t.Errorf("child %s/%s reported as dropped: %v", c.Type, c.Name, got) + } + } +} + +// TestGalleryFilterBlockStaysValid is the control that keeps the fix honest +// about WHY the datagrid case is wrong. `filter { … }` is correct on a gallery +// — the same widget property, under that widget's own keyword — so a rule that +// simply banned `filter` as a child would break the form the skills teach. +// Uses the REAL gallery definition, not a fixture. +func TestGalleryFilterBlockStaysValid(t *testing.T) { + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatal(err) + } + def, ok := reg.Get("GALLERY") + if !ok { + t.Skip("gallery definition not embedded in this build") + } + gallery := wdg("gallery", "g", + wdg("filter", "f", wdg("textfilter", "tf")), + // A gallery has a `template` catch-all, so an arbitrary widget in its + // body is placed rather than dropped. + wdg("actionbutton", "btn"), + ) + if got := droppedTypes(def, gallery); got != nil { + t.Errorf("gallery children reported as dropped: %v", got) + } +} + +// TestUnroutedRuleStaysQuietWhenItCannotKnow pins the conservative bail-outs. +// The check-time rule must never claim a drop that exec would not make, so each +// case where routing depends on something the definition does not show has to +// produce silence. +func TestUnroutedRuleStaysQuietWhenItCannotKnow(t *testing.T) { + child := wdg("filter", "f") + cases := []struct { + name string + def *WidgetDefinition + }{ + {"no definition at all", nil}, + {"skeleton definition", &WidgetDefinition{WidgetID: "x", MDLName: "X"}}, + { + // With no child slots, applyChildSlots returns early and the auto + // pass hands leftovers to the first widgets-typed property — which + // this function cannot see. + name: "object lists but no child slots", + def: &WidgetDefinition{WidgetID: "x", MDLName: "X", + ObjectLists: []ObjectListMapping{{PropertyKey: "columns", MDLContainer: "column"}}}, + }, + { + // `template` IS the catch-all (defaultSlotContainer). + name: "has a template slot", + def: &WidgetDefinition{WidgetID: "x", MDLName: "X", + ChildSlots: []ChildSlotMapping{{PropertyKey: "content", MDLContainer: "template"}}}, + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := droppedTypes(c.def, wdg("x", "w", child)); got != nil { + t.Errorf("reported %v, want silence", got) + } + }) + } +} + +// TestUnroutedMessageNamesThisWidgetsSpelling: the useful half of the +// diagnostic. An author who copied a gallery example needs the word DataGrid +// uses for the same property, not a list to search. +func TestUnroutedMessageNamesThisWidgetsSpelling(t *testing.T) { + def := datagridLikeDef() + msg := unroutedChildMessage(def, wdg("filter", "f")) + if !strings.Contains(msg, "`controlbar`") { + t.Errorf("message does not name DataGrid's spelling of the slot: %s", msg) + } + if !strings.Contains(msg, "dropped on write") { + t.Errorf("message does not say what would happen: %s", msg) + } + // Control: a child with no counterpart elsewhere gets no invented advice. + if m := unroutedChildMessage(def, wdg("actionbutton", "btn")); strings.Contains(m, "spelled") { + t.Errorf("invented an alternative spelling for a widget that has none: %s", m) + } +} + +// TestCheckAndExecAgreeOnUnroutedChildren is the anti-drift guard. The two +// halves are separate entry points — a linter rule and a writer refusal — and +// the failure they exist to prevent is precisely the two disagreeing: a check +// that passes and a write that drops, which is where this started. +func TestCheckAndExecAgreeOnUnroutedChildren(t *testing.T) { + def := datagridLikeDef() + for _, w := range []*ast.WidgetV3{ + wdg("datagrid", "dg", wdg("filter", "f")), + wdg("datagrid", "dg", wdg("column", "c")), + wdg("datagrid", "dg", wdg("controlbar", "cb")), + wdg("datagrid", "dg", wdg("actionbutton", "btn"), wdg("column", "c")), + wdg("datagrid", "dg"), + } { + checkFired := len(validateUnroutedChildren(w, def, "page X")) > 0 + execRefused := refuseUnroutedChildren(def, w) != nil + if checkFired != execRefused { + t.Errorf("check=%v exec=%v for children %v — the two must agree", + checkFired, execRefused, droppedTypes(def, w)) + } + } +} + +func TestUnroutedViolationIsAnError(t *testing.T) { + v := validateUnroutedChildren(wdg("datagrid", "dg", wdg("filter", "f")), datagridLikeDef(), "page X") + if len(v) != 1 { + t.Fatalf("got %d violations, want 1", len(v)) + } + if v[0].RuleID != "MDL-WIDGET29" { + t.Errorf("RuleID = %s, want MDL-WIDGET29", v[0].RuleID) + } + if v[0].Severity != linter.SeverityError { + t.Errorf("Severity = %s, want error — a dropped widget is not a style note", v[0].Severity) + } +} From 0c0bebc30dc917160a093f1abd6771b3cf3c2849 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 20:19:13 +0000 Subject: [PATCH 16/18] feat(widget-describe): name the widgets-typed slots inside an object-list item MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `describe widget` is where MDL-WIDGET26 and MDL-WIDGET29 send an author to find out what a widget's body takes, and where the quick reference and the syntax topics point too. For a Data Grid 2 it did not answer the question that prompted those rules. describeContainers looped an object list's ItemProperties and never its ItemSlots, so a column's two widgets-typed slots — `content` and `filter` — were invisible, even though the property dump higher up in the same output shows them as `widgets`. The generated MDL example had the matching hole: `column item1 (showContentAs: 'attribute')` with no body, so nothing suggested a column takes one. The result was worse than an omission. The only filter-shaped container it named was `controlbar` — the grid-WIDE filter bar — so an author asking this command where the column filter goes was steered to the one place that renders "Unable to get filter store. Check parent widget configuration." Now: column object list -> columns authorable items: showContentAs, attribute, dynamicText, … slot content -> content: any other widget in the item body slot filter -> filter: textfilter | numberfilter | datefilter | dropdownfilter and the example shows the column body with a filter in it. What matters is the second half of each slot line — how a widget REACHES the slot — because AcceptedChildTypes is exactly what makes the filter go in the column's own braces with no wrapper. Both the accepted types and which slot is the default are read from the engine (itemSlotAcceptedChildTypes, defaultItemSlotKey) rather than restated: a second copy of the routing rule is the #1036 defect one layer up, and a test compares the described list against the engine's table so they cannot drift. Measured against Mendix 11.13.0 with DataGrid 2 (Data Widgets) 3.4.0. Controls: a widget whose containers are plain child slots (Gallery, Timeline) reports no item slots; stubbing the loop fails all four tests with `got map[]`; and the generated example still passes its own parses-as-written guard, verified separately with `check -p` at 0 errors. go test ./... clean; make lint clean; make check-mdl 524 pass, 0 fail; make check-skill-mdl clean; make check-findings OK. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .claude/skills/mendix/custom-widgets/SKILL.md | 15 +- docs/01-project/MDL_QUICK_REFERENCE.md | 2 +- mdl/executor/widget_describe.go | 70 ++++++++++ .../widget_describe_item_slots_test.go | 131 ++++++++++++++++++ mdl/executor/widget_unrouted_children_test.go | 26 +++- 6 files changed, 238 insertions(+), 7 deletions(-) create mode 100644 mdl/executor/widget_describe_item_slots_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index ae0c373db..30407f7c8 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -596,3 +596,4 @@ {"area": "mdl/executor", "date": "2026-09-12", "symptom": "`describe microflow` -> `exec` across every microflow in a project produced one that mxbuild refuses: CE0709 \"Sequence flow is not accepted by origin or destination\", with no microflow named in the message. `mxcli check` passed, `exec` reported success, and the project still OPENED in Studio Pro — only a build caught it. 1 of 41 microflows in ako/TestApp", "cause": "A Mendix end event accepts exactly ONE incoming sequence flow — joining two paths is what a merge is for. An empty `on error … { }` handler inside a branch whose sibling also returns gave one two: the error-handler flow and the else branch's return both landed on the same end event. DESCRIBE emits the empty block because MDL cannot say \"the error flow rejoins the main path at a merge\", which is what the stored Studio Pro document does", "file": "`mdl/executor/cmd_microflows_builder_flows.go` (mergeOverConnectedEndEvents), called from `cmd_microflows_builder_graph.go` beside applyFlowCurves", "insight": "**Measure in/out degree per node when a graph-shaped document fails to build.** CE0709 names neither the microflow nor the edge, and the round-trip diff was ~90% layout, so reading it got nowhere; counting inbound flows per node found the single over-connected end event in one pass and named the two colliding edges. **The guard belongs in a post-pass, not at the creation site**: the two flows are emitted by unrelated builders in either order — here the error flow lands FIRST and the branch's flow arrives afterwards — so a check at either site sees nothing wrong. A first attempt at the wiring site built cleanly and changed nothing, which is the tell. `applyFlowCurves` was already there for the same reason and is the precedent to copy. **Hand-built AST did not reproduce it** — the synthetic version put the two paths on separate end events — so the test parses real MDL through `visitor.Build`; when a builder bug will not reproduce from a hand-made AST, that is evidence the visitor's output differs, not that the bug is imaginary. **Prefer the join Studio Pro writes**: a second end event would also satisfy the in-degree rule but needs its own return value, which MDL never stated. The two microflows that lose merges and still build clean are the control that in-degree 2, not merge loss, is the trigger", "refs": []} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "`check --references` / `exec` on a view entity reports `could not parse select clause from OQL query` for a query Mendix itself wrote — specifically anything in the FROM-first clause order (`from … group by … select …`), which is what Studio Pro stores and therefore what `DESCRIBE ENTITY` emits. Feeding a describe straight back to check fails, so describe → edit → exec is the one loop that breaks on this document type", "cause": "`extractSelectClause` found SELECT and scanned FORWARD for FROM to end the column list. In the from-first order the FROM is BEHIND the SELECT, so the scan ran off the end and returned \"\". The MDL grammar had accepted both orders all along (`oqlQueryTerm`, MDLCatalog.g4) — only the hand-written scanner had not", "file": "`mdl/executor/oql_type_inference.go` (`extractSelectClause` → `extractSelectClauseOK` + `topLevelKeywordIndex`; the new report in `ValidateOQLSyntax`)", "insight": "**An empty select clause is a checker-off switch, and that is the real finding.** The visible symptom was ONE bogus error from `inferOQLTypes`; the invisible half was that MDL030 (missing alias), MDL072 (quoted alias) and every MDL031 type rule sit behind `if selectClause != \"\"` and simply stopped running — a from-first view got NO OQL checking at all. So the fix is two things: read both orders, AND report an unreadable clause instead of skipping it, so a third clause shape cannot go quiet again. **This is a RECURRENCE at the same function**: bug 9b (2026-07-09, same symptom string, same double consequence) was a case-comparison slip making it return \"\" for every query. Same fault, different input shape, two months apart — when a function's failure mode is 'returns empty and everything downstream shrugs', fix the shrug, not just the input. Two traps in widening it: the two orders need DIFFERENT terminator sets (admitting ORDER/LIMIT in the select-first order cuts `select o.Limit as Limit from …` in half, so the order is decided by whether FROM precedes SELECT), and the scanner must skip quoted runs, because `select s.\"Order\" as OrderValue` is exactly the form mxcli tells users to write so a reserved word survives MxBuild (CE0174). `ORDER BY` is matched as a phrase, since a bare ORDER is an ordinary name. Repro `mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl` + the `.fail.mdl` sibling that proves MDL030 runs on this order; tests `validate_oql_clause_order_test.go`, each pairing the from-first query with its select-first twin as the control. Reported by ako/view-entity-examples FINDINGS §3", "rules": ["MDL030", "MDL031", "MDL072"], "refs": ["ako/view-entity-examples"]} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget written inside a pluggable widget's body is BUILT AND THROWN AWAY — `check` silent, `exec` reports success, the build is clean, and `DESCRIBE PAGE` showing the widget gone is the only symptom. Reported as a Data Grid 2 column filter written `COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (…) }` — the form `mxcli syntax page widgets` documented", "cause": "The four passes that place a pluggable widget's children — `applyChildSlots`, `applyObjectLists`, and two auto-discovery passes in `buildPluggable` — each `continue` past a child they do not recognise, and `applyChildSlots` parks the leftovers in `defaultWidgets`, which is assigned ONLY if the widget declares a `template` slot (`defaultSlotContainer`). A Gallery has one; DataGrid 2 does not, so its leftovers were discarded with no error. Not datagrid-specific: one `continue` per pass, shared by every pluggable widget", "file": "`mdl/executor/widget_unrouted_children.go` (new: `unroutedPluggableChildren`, `validateUnroutedChildren` = MDL-WIDGET29, `refuseUnroutedChildren`), wired in `mdl/executor/widget_engine.go` (`buildPluggable`) and `mdl/executor/validate_widgets.go`", "insight": "**Two independent mistakes were needed to hide this, and the docs supplied one of them.** `FILTER` is the GALLERY keyword for the filters placeholder — DataGrid 2 spells the same property `controlbar` (`widgetSlotKeywordOverrides`) — and `widgetV3` is `type name props? body?`, so a `filter` written AFTER the column's parentheses is not a block on the column at all but a SIBLING widget. The syntax topic showed exactly that, so the likeliest thing to write was the thing that vanished. **Derive the diagnostic's advice from the table the engine already reads**: `widgetSlotKeywordOverrides` knows both spellings of the same property, so the message can say 'on this widget that slot is spelled `controlbar`' rather than listing containers to hunt through. **Check and exec share one predicate** (`unroutedPluggableChildren`) with a test asserting they agree, because a checker and a writer disagreeing about a drop is how this started. The predicate is deliberately CONSERVATIVE — silent with no definition, with no child slots (the auto pass may still take the child), and when a `template` catch-all exists — so `check` can never claim a drop `exec` would not make. **Check MDL-WIDGET rule numbers before picking one**: there is no registry, 01–28 were taken, and the obvious next number collided with the repeatable-property rule. Sibling rule MDL-WIDGET26 covers the neighbouring case (a container KEYWORD in the same position); it could not report this one because a real widget resolves perfectly well on its own. Repro `mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl` — a plain `.mdl`, not `.fail.mdl`, because the rule needs the parent's definition and `make check-mdl` runs with no project (the Makefile's #891/#892 note). Tests `widget_unrouted_children_test.go`, control = the real Gallery definition, on which the same `filter { … }` block must stay valid. **Measured end to end on Mendix 11.13.0 with DataGrid 2 (Data Widgets) 3.4.0**, which matters because the earlier write-up called this unverifiable without a project — DataGrid 2 ships in every Mendix 11 app, so the real definition is always one `-p` away. With the guard STUBBED, `exec --no-check` printed `Created page` and exited 0 while `describe page` came back `column MeterCode (Attribute: MeterCode, Caption: 'Meter')` with no filter at all — the reported defect reproduced against a real model. With the guard in place the same script is refused (exit 1, naming `controlbar`), the column-braces form persists as `column … { textfilter tf }`, and `mx check` on the result is 0 errors. Reading the definition from the project's `.mpk` is also what keeps the rule current: upgrading Data Widgets from the Marketplace changes what the rule sees with no mxcli release. Reported by ako/view-entity-examples FINDINGS §2", "rules": ["MDL-WIDGET29"], "refs": ["ako/view-entity-examples"]} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "`describe widget datagrid -p app.mpr` — the command the MDL-WIDGET26/29 messages and the docs all point at for \"what does this widget's body take?\" — does not mention that a column can hold a filter. `Body containers` lists `column`, `controlbar`, `emptyplaceholder`, and the column's `items:` line lists only its scalar sub-properties", "cause": "`describeContainers` looped `ol.ItemProperties` and never `ol.ItemSlots`, so the two WIDGETS-typed slots of a DataGrid column (`content`, `filter`) were invisible — even though the property dump higher up in the same output shows them as `widgets`. The generated MDL example had the matching hole: it emitted `column item1 (showContentAs: 'attribute')` with no body, so nothing suggested a column takes one", "file": "`mdl/executor/widget_describe.go` (`DescribedItemSlot`, `describeContainers`, the container printer, and the object-list branch of the example generator)", "insight": "**An incomplete description was worse here than no description.** The only filter-shaped container it named was `controlbar` — the grid-WIDE filter bar — so an author asking this command where the column filter goes was steered to the one place that renders \"Unable to get filter store. Check parent widget configuration.\" That closes a loop with MDL-WIDGET29: the refusal names what the parent declares and says to look here, so the answer had to actually be here. **Say how a widget REACHES the slot, not just that the slot exists**: the useful line is `slot filter -> filter: textfilter | numberfilter | datefilter | dropdownfilter`, because AcceptedChildTypes is precisely what makes the filter go in the column's own braces with no wrapper. Both the accepted types and which slot is the default come from the engine (`itemSlotAcceptedChildTypes`, `defaultItemSlotKey`) rather than being restated — a second copy of the routing rule is the #1036 defect one layer up. Control: a widget whose containers are plain child slots (Gallery, Timeline) must report no item slots, and stubbing the loop makes all four tests fail with `got map[]`. Tests `widget_describe_item_slots_test.go`. Found by asking whether `describe widget` answered the question behind ako/view-entity-examples FINDINGS §2 — it did not", "refs": ["ako/view-entity-examples", "#1036"]} diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index 8b6b42d57..5e386c3c4 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -72,7 +72,20 @@ The error now names the container keyword and rewrites your entry into the form that works. `describe widget -p ` lists a widget's container keywords -under **Body containers**. +under **Body containers**, and — for an object list — the widgets-typed **slots +inside one item**, with the widget types that route into each: + +``` +column object list -> columns authorable + items: showContentAs, attribute, dynamicText, … + slot content -> content: any other widget in the item body + slot filter -> filter: textfilter | numberfilter | datefilter | dropdownfilter +``` + +Read that last line before guessing where something goes. It says a Data Grid 2 +column filter is written directly in the **column's** braces — not in +`controlbar`, which is the grid-wide filter bar and renders "Unable to get +filter store" if you put a column filter there. ### When the name is not found diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index f6143404c..7191f5832 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1341,7 +1341,7 @@ MDL uses explicit property declarations for pages: | Repeated widget entries | ` ( … )` **in the widget body** | A repeatable property (FileUploader `allowedFileFormats`, HTML Element `attributes`, a chart's `series`) is a block, never a property value. `attributes: [(attributeName: 'x')]` is **MDL-WIDGET27** — it used to check clean, exec, and vanish from storage. `describe widget -p app.mpr` lists the container keywords | | Data grid 2 column filter | `column c (attribute: A) { textfilter f }` | **Inside the column's braces.** `column c (…) filter f { … }` is the GALLERY form — the grammar reads it as a column with no body plus a sibling `filter` widget, which the grid has nowhere to put; it used to be dropped on write and is now **MDL-WIDGET29**. A grid-wide filter bar is `controlbar`; a gallery spells that same slot `filter`. Match the filter to the column's type (String → `textfilter`, number → `numberfilter`, DateTime → `datefilter`, Enumeration → `dropdownfilter`, Boolean → none) | | Widget with nowhere to go | any widget in a pluggable widget's body | A child matching no container, slot or `template` catch-all is **MDL-WIDGET29** at check time and refused by `exec`. `describe widget -p app.mpr` lists what the parent declares. Needs the parent's definition, so it is silent without `-p` | -| Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | +| Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. **Body containers** names what the widget's body takes, and for an object list the widgets-typed slots *inside one item* plus the widget types that route into each — that is where `column … { textfilter }` is spelled out. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | | Variable binding | `datasource: $Var` | `dataview dv (datasource: $Product) { ... }` | diff --git a/mdl/executor/widget_describe.go b/mdl/executor/widget_describe.go index 06b672fd5..3f95962b3 100644 --- a/mdl/executor/widget_describe.go +++ b/mdl/executor/widget_describe.go @@ -169,6 +169,20 @@ type DescribedContainer struct { // would be the same mistake one layer up. Authorable bool `json:"authorable"` + // ItemSlots lists the WIDGETS-typed slots inside one item of an object list + // — a DataGrid column's `content` and `filter`, an Accordion group's + // `headerContent`. + // + // These were missing, and their absence had teeth. `items` above lists only + // the item's scalar sub-properties, so `describe widget datagrid` named + // `controlbar` as the one filter-shaped thing a grid declares and said + // nothing about a column taking a filter at all. An author asking this + // command where the column filter goes was therefore steered to the + // grid-wide filter bar — which renders "Unable to get filter store" at + // runtime (ako/view-entity-examples FINDINGS §2). A description that omits + // the right answer and offers a wrong-looking one is worse than silence. + ItemSlots []DescribedItemSlot `json:"itemSlots,omitempty"` + // items carries each sub-property's writable value as the WIDGET DEFINITION // records it — the mapping's `default`/`value` and its enumValues. Unexported, // so the JSON shape is unchanged. @@ -181,6 +195,18 @@ type DescribedContainer struct { items []DescribedProperty } +// DescribedItemSlot is one widgets-typed slot inside an object-list item, with +// the two ways a widget reaches it: an explicit ` { … }` block, or — +// for the types in Accepts — being written directly in the item's body. +type DescribedItemSlot struct { + Keyword string `json:"keyword"` + PropertyKey string `json:"propertyKey"` + Accepts []string `json:"accepts,omitempty"` + // Default marks the slot that takes any child matching neither route + // (defaultItemSlotKey). Exactly one slot per object list has it. + Default bool `json:"default,omitempty"` +} + func resolveWidgetTarget(registry *WidgetRegistry, arg string) (string, *WidgetDefinition) { if strings.Contains(arg, ".") { if def, ok := registry.GetByWidgetID(arg); ok { @@ -461,6 +487,19 @@ func PrintWidgetDescription(out io.Writer, d WidgetDescription) { if len(c.ItemKeys) > 0 { fmt.Fprintf(out, " %-34s items: %s\n", "", strings.Join(c.ItemKeys, ", ")) } + for _, is := range c.ItemSlots { + // Say how a widget REACHES the slot, not just that it exists — + // the accepted types are the answer to "where does the column + // filter go?", and they are the reason a filter belongs in the + // column's own braces rather than in the grid's filter bar. + how := "any other widget in the item body" + if len(is.Accepts) > 0 { + how = strings.Join(is.Accepts, " | ") + } else if !is.Default { + how = is.Keyword + " { … }" + } + fmt.Fprintf(out, " %-34s slot %s -> %s: %s\n", "", is.Keyword, is.PropertyKey, how) + } } } } @@ -557,6 +596,18 @@ func describeContainers(def *WidgetDefinition) []DescribedContainer { Key: ip.PropertyKey, Type: ip.Operation, Default: def, Enum: ip.EnumValues, }) } + // Ask the engine which slot is the default rather than restating the + // "content, else the first one" rule — a second copy of that rule is + // how the container lists drifted apart in #1036. + defaultKey := defaultItemSlotKey(&ol) + for _, is := range ol.ItemSlots { + c.ItemSlots = append(c.ItemSlots, DescribedItemSlot{ + Keyword: strings.ToLower(is.MDLContainer), + PropertyKey: is.PropertyKey, + Accepts: append([]string(nil), is.AcceptedChildTypes...), + Default: is.PropertyKey == defaultKey, + }) + } out = append(out, c) } sort.Slice(out, func(i, j int) bool { return out[i].Keyword < out[j].Keyword }) @@ -659,6 +710,25 @@ func buildUsageExample(d WidgetDescription) (example string, omitted []string) { if k, lit := itemExampleLiteral(d, c); k != "" { item += " (" + k + ": " + lit + ")" } + // Show the item's own body when a slot routes a widget type INTO it. + // That routing is the part nobody guesses — a DataGrid column's filter + // goes in the column's braces, and an example that stops at the closing + // paren leaves the reader thinking a column has no body at all. Which is + // how the gallery `filter { … }` form got written on a grid instead. + var inner []string + for _, is := range c.ItemSlots { + if len(is.Accepts) == 0 { + continue + } + n++ + inner = append(inner, fmt.Sprintf(" %s item%d -- routed to this entry's `%s`", + is.Accepts[0], n, is.PropertyKey)) + } + if len(inner) > 0 { + body = append(body, item+" { -- one entry of `"+c.PropertyKey+"`\n"+ + strings.Join(inner, "\n")+"\n }") + continue + } body = append(body, item+" -- one entry of `"+c.PropertyKey+"`") } diff --git a/mdl/executor/widget_describe_item_slots_test.go b/mdl/executor/widget_describe_item_slots_test.go new file mode 100644 index 000000000..e86348c7a --- /dev/null +++ b/mdl/executor/widget_describe_item_slots_test.go @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" +) + +// `describe widget` is where an author is sent to find out what a widget's body +// takes — the MDL-WIDGET26/29 messages say so, and so do the quick reference and +// the syntax topics. It listed a widget's containers and, for an object list, +// that item's SCALAR sub-properties. It did not list the item's widgets-typed +// SLOTS. +// +// For a Data Grid 2 that omission had teeth. The only filter-shaped thing it +// named was `controlbar` — the grid-wide filter bar — while `column`'s own +// `filter` slot, the actual home of a column filter, went unmentioned. So the +// command answered "where does the filter go?" with the one place that renders +// "Unable to get filter store" at runtime (ako/view-entity-examples FINDINGS §2). +// +// These tests hold the answer in place. + +func columnSlots(t *testing.T) map[string]DescribedItemSlot { + t.Helper() + var col *DescribedContainer + for _, c := range describeContainers(datagridLikeDef()) { + if c.Keyword == "column" { + cc := c + col = &cc + } + } + if col == nil { + t.Fatal("no `column` container described for a datagrid-shaped definition") + } + out := map[string]DescribedItemSlot{} + for _, s := range col.ItemSlots { + out[s.PropertyKey] = s + } + return out +} + +func TestDescribeWidget_NamesTheSlotsInsideAnObjectListItem(t *testing.T) { + slots := columnSlots(t) + if _, ok := slots["filter"]; !ok { + t.Fatalf("a column's `filter` slot is not described; got %v", slots) + } + if _, ok := slots["content"]; !ok { + t.Errorf("a column's `content` slot is not described; got %v", slots) + } +} + +// The accepted types are the useful half — they are what tells the reader the +// filter is written directly in the column's braces rather than in a nested +// block. Compared against the engine's own table, so the description cannot +// drift from the routing it describes. +func TestDescribeWidget_ItemSlotAcceptedTypesComeFromTheEngineTable(t *testing.T) { + want := itemSlotAcceptedChildTypes["com.mendix.widget.web.datagrid.Datagrid"]["columns"]["filter"] + if len(want) == 0 { + t.Fatal("the engine table no longer routes any widget type into a column's filter slot — " + + "if that is deliberate, this test and the description have to follow it") + } + got := columnSlots(t)["filter"].Accepts + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Errorf("described accepts = %v, engine routes %v", got, want) + } +} + +// Exactly one slot per item is the default, and which one is decided by the +// engine (defaultItemSlotKey), not restated here. +func TestDescribeWidget_MarksTheDefaultItemSlot(t *testing.T) { + slots := columnSlots(t) + var defaults []string + for key, s := range slots { + if s.Default { + defaults = append(defaults, key) + } + } + if len(defaults) != 1 { + t.Fatalf("want exactly one default slot, got %v", defaults) + } + if defaults[0] != "content" { + t.Errorf("default slot = %s, want content", defaults[0]) + } +} + +// Control: a widget whose containers are plain child slots must report no item +// slots at all — the new field cannot start inventing them. +func TestDescribeWidget_ChildSlotsHaveNoItemSlots(t *testing.T) { + for _, c := range describeContainers(datagridLikeDef()) { + if c.Kind == "child slot" && len(c.ItemSlots) > 0 { + t.Errorf("child slot %s reported item slots %v", c.Keyword, c.ItemSlots) + } + } + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatal(err) + } + def, ok := reg.Get("GALLERY") + if !ok { + t.Skip("gallery definition not embedded in this build") + } + for _, c := range describeContainers(def) { + if len(c.ItemSlots) > 0 { + t.Errorf("gallery container %s reported item slots %v — it has no object lists", + c.Keyword, c.ItemSlots) + } + } +} + +func TestPrintWidgetDescription_SaysHowToReachAnItemSlot(t *testing.T) { + var buf bytes.Buffer + PrintWidgetDescription(&buf, WidgetDescription{ + WidgetID: "com.mendix.widget.web.datagrid.Datagrid", + MDLName: "DATAGRID", + Kind: "pluggable", + Containers: describeContainers(datagridLikeDef()), + }) + out := buf.String() + if !strings.Contains(out, "slot filter -> filter:") { + t.Errorf("rendered output does not name the column's filter slot:\n%s", out) + } + if !strings.Contains(out, "textfilter") { + t.Errorf("rendered output does not say which widgets route into it:\n%s", out) + } + // The default slot reads as a sentence, not as an empty accepts list. + if !strings.Contains(out, "any other widget in the item body") { + t.Errorf("rendered output does not explain the default slot:\n%s", out) + } +} diff --git a/mdl/executor/widget_unrouted_children_test.go b/mdl/executor/widget_unrouted_children_test.go index a5ac4a45b..f05ce4392 100644 --- a/mdl/executor/widget_unrouted_children_test.go +++ b/mdl/executor/widget_unrouted_children_test.go @@ -27,12 +27,28 @@ func datagridLikeDef() *WidgetDefinition { Operation: "widgets", } } + // The column's two widgets-typed item slots, with the accepted child types + // read from the engine's own table rather than retyped — makeObjectListMapping + // builds the real definition the same way. + itemSlot := func(key string) ItemSlotMapping { + return ItemSlotMapping{ + PropertyKey: key, + MDLContainer: strings.ToUpper(key), + Operation: "widgets", + AcceptedChildTypes: itemSlotAcceptedChildTypes[id]["columns"][key], + } + } return &WidgetDefinition{ - WidgetID: id, - MDLName: "DATAGRID", - WidgetKind: "pluggable", - ObjectLists: []ObjectListMapping{{PropertyKey: "columns", MDLContainer: "column"}}, - ChildSlots: []ChildSlotMapping{slot("emptyPlaceholder"), slot("filtersPlaceholder")}, + WidgetID: id, + MDLName: "DATAGRID", + WidgetKind: "pluggable", + ObjectLists: []ObjectListMapping{{ + PropertyKey: "columns", + MDLContainer: "column", + ItemProperties: []ItemPropertyMapping{{PropertyKey: "attribute", Operation: "attribute"}}, + ItemSlots: []ItemSlotMapping{itemSlot("content"), itemSlot("filter")}, + }}, + ChildSlots: []ChildSlotMapping{slot("emptyPlaceholder"), slot("filtersPlaceholder")}, } } From 6c12702423594d97347d88215492c05f6f456ec1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 21:00:55 +0000 Subject: [PATCH 17/18] feat(view-entity): give a view entity its association to a persistent entity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Selecting a persistent entity's id under an alias gives a view entity an ASSOCIATION to that entity, named after the alias. The id column is not one of the view entity's attributes, and there is no separate declaration anywhere: Studio Pro creates the member when the column is added, so the column IS the declaration. mxcli modelled the select list as attributes only, and got three things wrong at once (ako/view-entity-examples FINDINGS §1): 1. `check` aligns columns with declared attributes BY POSITION and did not skip the id column — so it did not merely go unchecked, it SHIFTED every attribute after it onto its neighbour's expression. With the id first: "attribute 'TotalKwh': declared as Decimal but OQL expression returns Integer". With it last: "OQL select has 4 columns but 3 attributes declared". Neither describes the script. 2. `exec` created no association member, so the OQL and the model disagreed — CE1613 on 11.13/11.14, CE6770 on 10.24. 3. Adding one by hand wrote `Source: null`, which mxbuild refuses with CE6771 "It is not possible to create associations to/from View Entities". The measurement came first, before any Go. Build the broken model with mxcli, apply the reporter's python patch that adds only "Source": { "$Type": "DomainModels$OqlViewAssociationSource", "Reference": "" } and re-run mx check: 2 errors to 0 on 11.13.0. That pinned the target shape and showed that ONE field clears both errors, because the association is also what makes the id column legal on the view entity. So the association is derived from the OQL rather than given new syntax, which is also what makes describe → exec round-trip: the OQL carries the column, so a described view entity rebuilds its association with no second statement to keep in step. A plain `create association` with a view entity at either end is now refused at check AND exec time, pointing at the column form instead of the pre-fix workaround. The alias must be free in the module, case-insensitively — Mendix's "Duplicate name" rule, which the reporter found the hard way. Three things the report did not say, found by running its own example: - It calls the stored element a CrossAssociation. That is true only when the target is in another module; same-module is a plain Association. Both needed the field, so a fix pinned to one would have worked for one project layout. - The read path is not optional. `create or modify association` round-trips through the semantic model and wrote Source back as null, silently converting a working project into CE6771. (The ALTER path does not, because it gen-mutates the stored document — so the loss is narrower than the report implies, but real. A/B'd both ways.) - `extractAliasMap` matched neither `join r/Mod.Assoc/Mod.Target as m` nor `from Mappings."Order" as o` — an association-path join and a quoted reserved word, which between them are the reporter's literal example. The alias resolved to nothing and no association was created at all. `cast(m.ID as string) as MeterId` is deliberately NOT an association: it is a legitimate flat design (one query instead of two, no objects materialised in the client), so detection is the bare `.ID` form only, with a control test. Also collapses scriptContext's two parallel collectors into one. They were kept in step by hand and were not: collectSingle had no CreateConstantStmt case, and adding view-entity tracking to one left the other silent — so the new CE6771 rule passed against a project and did nothing on a one-script repro, which is the ordinary shape. Measured on Mendix 11.13.0, both engines, 0 errors: the reported same-module case, the reporter's cross-module quoted-entity case, a describe → exec into a fresh project, and a re-run (Unchanged). Controls: the pre-fix binary on the same script gives 2 × CE1613, and the pre-fix binary drops the patched Source on rewrite. go test ./... clean; make lint clean; make check-mdl 525 pass, 0 fail; make check-skill-mdl clean; make check-findings OK. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../fix-issue/findings/mdl-executor.jsonl | 2 + .../skills/mendix/write-oql-queries/SKILL.md | 44 ++ docs/01-project/MDL_QUICK_REFERENCE.md | 1 + .../bug-tests/view-entity-association.mdl | 116 +++++ .../modelsdk/association_move_write.go | 10 + mdl/backend/modelsdk/domainmodel.go | 13 + mdl/backend/modelsdk/domainmodel_write.go | 12 + mdl/executor/cmd_associations.go | 13 + mdl/executor/cmd_entities.go | 24 + mdl/executor/oql_type_inference.go | 54 ++- mdl/executor/oql_view_associations.go | 422 ++++++++++++++++++ mdl/executor/oql_view_associations_test.go | 138 ++++++ mdl/executor/validate.go | 105 ++--- sdk/domainmodel/domainmodel.go | 47 +- sdk/mpr/parser_domainmodel.go | 16 + sdk/mpr/writer_domainmodel.go | 25 +- sdk/mpr/writer_domainmodel_test.go | 63 +++ 17 files changed, 1030 insertions(+), 75 deletions(-) create mode 100644 mdl-examples/bug-tests/view-entity-association.mdl create mode 100644 mdl/executor/oql_view_associations.go create mode 100644 mdl/executor/oql_view_associations_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 30407f7c8..bfbec4a65 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -597,3 +597,5 @@ {"area": "mdl/executor", "date": "2026-09-12", "symptom": "`check --references` / `exec` on a view entity reports `could not parse select clause from OQL query` for a query Mendix itself wrote — specifically anything in the FROM-first clause order (`from … group by … select …`), which is what Studio Pro stores and therefore what `DESCRIBE ENTITY` emits. Feeding a describe straight back to check fails, so describe → edit → exec is the one loop that breaks on this document type", "cause": "`extractSelectClause` found SELECT and scanned FORWARD for FROM to end the column list. In the from-first order the FROM is BEHIND the SELECT, so the scan ran off the end and returned \"\". The MDL grammar had accepted both orders all along (`oqlQueryTerm`, MDLCatalog.g4) — only the hand-written scanner had not", "file": "`mdl/executor/oql_type_inference.go` (`extractSelectClause` → `extractSelectClauseOK` + `topLevelKeywordIndex`; the new report in `ValidateOQLSyntax`)", "insight": "**An empty select clause is a checker-off switch, and that is the real finding.** The visible symptom was ONE bogus error from `inferOQLTypes`; the invisible half was that MDL030 (missing alias), MDL072 (quoted alias) and every MDL031 type rule sit behind `if selectClause != \"\"` and simply stopped running — a from-first view got NO OQL checking at all. So the fix is two things: read both orders, AND report an unreadable clause instead of skipping it, so a third clause shape cannot go quiet again. **This is a RECURRENCE at the same function**: bug 9b (2026-07-09, same symptom string, same double consequence) was a case-comparison slip making it return \"\" for every query. Same fault, different input shape, two months apart — when a function's failure mode is 'returns empty and everything downstream shrugs', fix the shrug, not just the input. Two traps in widening it: the two orders need DIFFERENT terminator sets (admitting ORDER/LIMIT in the select-first order cuts `select o.Limit as Limit from …` in half, so the order is decided by whether FROM precedes SELECT), and the scanner must skip quoted runs, because `select s.\"Order\" as OrderValue` is exactly the form mxcli tells users to write so a reserved word survives MxBuild (CE0174). `ORDER BY` is matched as a phrase, since a bare ORDER is an ordinary name. Repro `mdl-examples/bug-tests/view-entity-oql-from-first-clause-order.mdl` + the `.fail.mdl` sibling that proves MDL030 runs on this order; tests `validate_oql_clause_order_test.go`, each pairing the from-first query with its select-first twin as the control. Reported by ako/view-entity-examples FINDINGS §3", "rules": ["MDL030", "MDL031", "MDL072"], "refs": ["ako/view-entity-examples"]} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "A widget written inside a pluggable widget's body is BUILT AND THROWN AWAY — `check` silent, `exec` reports success, the build is clean, and `DESCRIBE PAGE` showing the widget gone is the only symptom. Reported as a Data Grid 2 column filter written `COLUMN c (Attribute: A) FILTER f { TEXTFILTER tf (…) }` — the form `mxcli syntax page widgets` documented", "cause": "The four passes that place a pluggable widget's children — `applyChildSlots`, `applyObjectLists`, and two auto-discovery passes in `buildPluggable` — each `continue` past a child they do not recognise, and `applyChildSlots` parks the leftovers in `defaultWidgets`, which is assigned ONLY if the widget declares a `template` slot (`defaultSlotContainer`). A Gallery has one; DataGrid 2 does not, so its leftovers were discarded with no error. Not datagrid-specific: one `continue` per pass, shared by every pluggable widget", "file": "`mdl/executor/widget_unrouted_children.go` (new: `unroutedPluggableChildren`, `validateUnroutedChildren` = MDL-WIDGET29, `refuseUnroutedChildren`), wired in `mdl/executor/widget_engine.go` (`buildPluggable`) and `mdl/executor/validate_widgets.go`", "insight": "**Two independent mistakes were needed to hide this, and the docs supplied one of them.** `FILTER` is the GALLERY keyword for the filters placeholder — DataGrid 2 spells the same property `controlbar` (`widgetSlotKeywordOverrides`) — and `widgetV3` is `type name props? body?`, so a `filter` written AFTER the column's parentheses is not a block on the column at all but a SIBLING widget. The syntax topic showed exactly that, so the likeliest thing to write was the thing that vanished. **Derive the diagnostic's advice from the table the engine already reads**: `widgetSlotKeywordOverrides` knows both spellings of the same property, so the message can say 'on this widget that slot is spelled `controlbar`' rather than listing containers to hunt through. **Check and exec share one predicate** (`unroutedPluggableChildren`) with a test asserting they agree, because a checker and a writer disagreeing about a drop is how this started. The predicate is deliberately CONSERVATIVE — silent with no definition, with no child slots (the auto pass may still take the child), and when a `template` catch-all exists — so `check` can never claim a drop `exec` would not make. **Check MDL-WIDGET rule numbers before picking one**: there is no registry, 01–28 were taken, and the obvious next number collided with the repeatable-property rule. Sibling rule MDL-WIDGET26 covers the neighbouring case (a container KEYWORD in the same position); it could not report this one because a real widget resolves perfectly well on its own. Repro `mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl` — a plain `.mdl`, not `.fail.mdl`, because the rule needs the parent's definition and `make check-mdl` runs with no project (the Makefile's #891/#892 note). Tests `widget_unrouted_children_test.go`, control = the real Gallery definition, on which the same `filter { … }` block must stay valid. **Measured end to end on Mendix 11.13.0 with DataGrid 2 (Data Widgets) 3.4.0**, which matters because the earlier write-up called this unverifiable without a project — DataGrid 2 ships in every Mendix 11 app, so the real definition is always one `-p` away. With the guard STUBBED, `exec --no-check` printed `Created page` and exited 0 while `describe page` came back `column MeterCode (Attribute: MeterCode, Caption: 'Meter')` with no filter at all — the reported defect reproduced against a real model. With the guard in place the same script is refused (exit 1, naming `controlbar`), the column-braces form persists as `column … { textfilter tf }`, and `mx check` on the result is 0 errors. Reading the definition from the project's `.mpk` is also what keeps the rule current: upgrading Data Widgets from the Marketplace changes what the rule sees with no mxcli release. Reported by ako/view-entity-examples FINDINGS §2", "rules": ["MDL-WIDGET29"], "refs": ["ako/view-entity-examples"]} {"area": "mdl/executor", "date": "2026-09-12", "symptom": "`describe widget datagrid -p app.mpr` — the command the MDL-WIDGET26/29 messages and the docs all point at for \"what does this widget's body take?\" — does not mention that a column can hold a filter. `Body containers` lists `column`, `controlbar`, `emptyplaceholder`, and the column's `items:` line lists only its scalar sub-properties", "cause": "`describeContainers` looped `ol.ItemProperties` and never `ol.ItemSlots`, so the two WIDGETS-typed slots of a DataGrid column (`content`, `filter`) were invisible — even though the property dump higher up in the same output shows them as `widgets`. The generated MDL example had the matching hole: it emitted `column item1 (showContentAs: 'attribute')` with no body, so nothing suggested a column takes one", "file": "`mdl/executor/widget_describe.go` (`DescribedItemSlot`, `describeContainers`, the container printer, and the object-list branch of the example generator)", "insight": "**An incomplete description was worse here than no description.** The only filter-shaped container it named was `controlbar` — the grid-WIDE filter bar — so an author asking this command where the column filter goes was steered to the one place that renders \"Unable to get filter store. Check parent widget configuration.\" That closes a loop with MDL-WIDGET29: the refusal names what the parent declares and says to look here, so the answer had to actually be here. **Say how a widget REACHES the slot, not just that the slot exists**: the useful line is `slot filter -> filter: textfilter | numberfilter | datefilter | dropdownfilter`, because AcceptedChildTypes is precisely what makes the filter go in the column's own braces with no wrapper. Both the accepted types and which slot is the default come from the engine (`itemSlotAcceptedChildTypes`, `defaultItemSlotKey`) rather than being restated — a second copy of the routing rule is the #1036 defect one layer up. Control: a widget whose containers are plain child slots (Gallery, Timeline) must report no item slots, and stubbing the loop makes all four tests fail with `got map[]`. Tests `widget_describe_item_slots_test.go`. Found by asking whether `describe widget` answered the question behind ako/view-entity-examples FINDINGS §2 — it did not", "refs": ["ako/view-entity-examples", "#1036"]} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "A view entity cannot be given an association to a persistent entity. Three separate failures: (a) `mxcli check` misreports types — with the id column first, \"attribute 'TotalKwh': declared as Decimal but OQL expression returns Integer\"; with it last, \"OQL select has 4 columns but 3 attributes declared\"; (b) `exec` writes the view entity and creates NO association member, so the build fails CE1613 \"The selected association 'X' no longer exists\" (11.13/11.14) or CE6770 \"View Entity is out of sync with the OQL Query\" (10.24); (c) adding it by hand with `create association` gives CE6771 \"It is not possible to create associations to/from View Entities\"", "cause": "Selecting a persistent entity's id under an alias gives a view entity an ASSOCIATION whose name is the alias — the column IS the declaration, and Studio Pro creates the member when the column is added. mxcli modelled the select list as attributes only. (a) columns are aligned with declared attributes BY POSITION, so an unrecognised id column SHIFTS every attribute after it; (b) nothing created the member; (c) the association Studio Pro writes carries `Source: {$Type: DomainModels$OqlViewAssociationSource, Reference: }` and mxcli wrote `Source: null` — the `default:` arm of the source switch in both engines' association serializers", "file": "`mdl/executor/oql_view_associations.go` (new), `mdl/executor/cmd_entities.go` (execCreateViewEntity), `mdl/executor/oql_type_inference.go` (extractAliasMap, the two alignment sites), `mdl/executor/cmd_associations.go` + `validate.go` (CE6771 refusal), `sdk/domainmodel/domainmodel.go` (OqlViewAssociationSource, Association/CrossModuleAssociation.ViewSourceReference), `sdk/mpr/{parser,writer}_domainmodel.go`, `mdl/backend/modelsdk/{domainmodel,domainmodel_write,association_move_write}.go`", "insight": "**One field clears two errors, and measuring that first was the whole plan.** Before writing any Go: build the broken model with mxcli, apply the reporter's python patch that adds only the Source subdocument, re-run `mx check` — 2 errors (CE6771 + CE6770) to 0 on 11.13.0. That proves the target shape AND proves the association is also what makes the id column legal on the view entity, which no amount of reading would have settled. **Derive the association from the OQL rather than inventing syntax**: the column is the declaration, so describe → exec round-trips with no second statement to keep in step (verified: describe into a fresh project, 0 errors). **Correction to the report**: it says Studio Pro writes a `CrossAssociation`; that is true only when the target is in ANOTHER module — same-module is a plain `Association`, and both needed the field, so a fix pinned to CrossAssociation alone would have worked for one project layout and not the other. **The read path is not optional**: `create or modify association` round-trips through the semantic model and wrote `Source: null` back, silently converting a working project into CE6771 — A/B'd (pre-fix `Source = null`, fixed `Unchanged association`). The ALTER path does NOT lose it, because it gen-mutates the stored document, so the loss is narrower than the report implies but real. **Two alias-resolution gaps found by running the reporter's own example**: `extractAliasMap` matched neither `join r/Mod.Assoc/Mod.Target as m` (association-path join — the ordinary way to reach a related entity) nor `from Mappings.\"Order\" as o` (a quoted reserved word — the only way to write that entity), so the reported case resolved to nothing and produced no association at all. Run the reporter's literal example, not a tidied one. **`cast(m.ID as string)` must NOT be treated as an association** — it is a legitimate flat design (one query instead of two, no objects materialised in the client), so detection is the bare `.ID` form only. Measured on 11.13.0, both engines, 0 errors; controls: pre-fix binary on the same script gives 2 × CE1613, and re-running is `Unchanged`. Example `mdl-examples/bug-tests/view-entity-association.mdl`; tests `oql_view_associations_test.go`, `sdk/mpr/writer_domainmodel_test.go`. Reported by ako/view-entity-examples FINDINGS §1", "ce": ["CE6770", "CE6771", "CE1613"], "refs": ["ako/view-entity-examples"]} +{"area": "mdl/executor", "date": "2026-09-12", "symptom": "A rule that consults the script's own definitions fires for a project object but not for one the SAME script creates — e.g. the CE6771 \"no association to a view entity\" check passed on a script that created the view entity and the association together, which is the ordinary shape", "cause": "`scriptContext` had TWO parallel collectors — `collectDefinitions` (whole program) and `collectSingle` (one statement) — switching over the same statement types and kept in step by hand. They were already out of step before this fix: `collectSingle` had no `CreateConstantStmt` case. Adding view-entity tracking to one left the other silent", "file": "`mdl/executor/validate.go` (`collectDefinitions` is now a loop over `collectSingle`)", "insight": "**Two lists that must agree are one list waiting to happen** — the third occurrence of this shape in the executor (see the `Associations`/`CrossAssociations` findings). The tell is a switch statement duplicated with a different signature; diff them before adding a case, because the drift is already there. Collapsing `collectDefinitions` into a loop over `collectSingle` removed the class, and picked up the pre-existing constants gap for free. Found because a new rule tested fine against a project and silently did nothing in a one-script repro", "refs": ["ako/view-entity-examples"]} diff --git a/.claude/skills/mendix/write-oql-queries/SKILL.md b/.claude/skills/mendix/write-oql-queries/SKILL.md index 09c9d1633..a60773c27 100644 --- a/.claude/skills/mendix/write-oql-queries/SKILL.md +++ b/.claude/skills/mendix/write-oql-queries/SKILL.md @@ -374,6 +374,50 @@ Do not rewrite a described view into select-first just to make it look familiar — the stored text is what MxBuild validates against, and a needless rewrite is a diff for nothing. +### Step 1c: Selecting an id makes an ASSOCIATION, not an attribute + +Selecting a persistent entity's `ID` under an alias gives the view entity an +association to that entity. The alias becomes the association's name, and the +column is **not** one of the view entity's attributes — so do not declare one +for it: + +```sql +create view entity Sales.OrdersVE ( + order_date: DateTime -- one attribute… +) as ( + from Sales."Order" as o + select o.ID as persistent_order -- …but two columns + , o.OrderDate as order_date +); +``` + +mxcli creates the association member from that column. There is no separate +statement for it, and `create association` with a view entity at either end is +refused — Mendix rejects it (CE6771), because the association needs an +`OqlViewAssociationSource` that a plain one does not have. + +Two rules: + +- **The alias must be free in the module, case-insensitively.** It is the + association's name, and Mendix reports *"Duplicate name 'Meter' in module + 'Trends'. Entities, associations and enumerations cannot share names."* So + `as meter` beside an entity called `Meter` fails — name it `MeterRef`. +- **Reach the target through a join if it is not the FROM entity**, and select + the id off *that* alias: `join r/Trends.Reading_Meter/Trends.Meter as m … select m.ID as MeterRef`. + +**Consider the flat alternative first.** An association costs a second query at +runtime — the view returns the foreign key, and the client then fetches the +referenced objects in a batched `IN (...)` per page, materialising real objects +in its state. Selecting a string copy instead is one statement, one join, no +second retrieve, and the id is still there to look the object up with: + +```sql +select cast(m.ID as string) as MeterId, m.MeterCode as MeterCode, … +``` + +Use the association when you want to bind widgets over it (`MeterRef/MeterCode`); +use the cast when you just need the value. + ### Step 2: Write SELECT Clause - Use **lowercase** aggregate functions: `sum()`, `avg()`, `count()` - Use `count(entity.ID)` not `count(*)` diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 7191f5832..895750e74 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -63,6 +63,7 @@ create persistent entity Module.Photo ( | Create with auditing | `create persistent entity Module.Name (attrs, owner: autoowner, ChangedBy: autochangedby, CreatedDate: autocreateddate, ChangedDate: autochangeddate);` | Pseudo-types like AutoNumber | | Create view entity | `create view entity Module.Name (attrs) as select ...;` | OQL-backed read-only | | View entity clause order | `... as select … from …;` **or** `... as from … group by … select …;` | Both are Mendix OQL and both are checked. The second is what **Studio Pro stores**, so it is what `DESCRIBE ENTITY` emits — describe → edit → exec round-trips. The declared attributes are matched to the select columns **by position**, in either order | +| View entity → persistent entity | `select t.ID as MyRef, …` in the OQL | Selecting the target's **id** under an alias gives the view entity an **association** named after the alias. It is not an attribute and gets no declaration: the column *is* the declaration, so mxcli creates the member (with the `OqlViewAssociationSource` mxbuild requires — without it, CE6771 + CE6770). A plain `create association` with a view entity at either end is **refused**. The alias must be free in the module, case-insensitively. `cast(t.ID as string) as MyId` is a plain String attribute instead — one query rather than two, no objects in the client | | Create external entity | `create external entity Module.Name from odata client Module.Client (...) (attrs);` | From consumed OData | | Create external entities | `create [or modify] external entities from Module.Client [into module] [entities (...)];` | Bulk from $metadata | | Drop entity | `drop entity Module.Name;` | | diff --git a/mdl-examples/bug-tests/view-entity-association.mdl b/mdl-examples/bug-tests/view-entity-association.mdl new file mode 100644 index 000000000..656c94883 --- /dev/null +++ b/mdl-examples/bug-tests/view-entity-association.mdl @@ -0,0 +1,116 @@ +-- Bug: a view entity could not be given an association to a persistent entity, +-- and mxcli got it wrong in three different places at once. +-- +-- Selecting a persistent entity's ID under an alias gives a view entity an +-- ASSOCIATION to that entity. The alias becomes the association's name, and the +-- id column is NOT one of the view entity's attributes. Studio Pro creates the +-- member when the column is added; there is no separate declaration anywhere. +-- +-- Reported by ako/view-entity-examples FINDINGS §1, measured there on +-- 10.24.24.119653 and 11.14.0. Re-measured here on 11.13.0. +-- +-- What went wrong, in order: +-- +-- 1. `mxcli check` aligned select columns with declared attributes BY +-- POSITION and did not skip the id column, so it did not merely go +-- unchecked — it SHIFTED every attribute after it onto its neighbour's +-- expression. With the id first: "attribute 'TotalKwh': declared as +-- Decimal but OQL expression returns Integer". With it last: "OQL select +-- has 4 columns but 3 attributes declared". Neither describes the script. +-- +-- 2. `exec` wrote the view entity and created NO association member, so the +-- OQL and the model disagreed: +-- 11.13/11.14: CE1613 "The selected association 'X' no longer exists." +-- 10.24: CE6770 "View Entity is out of sync with the OQL Query." +-- +-- 3. Adding it by hand with `create association` wrote one with `Source: null`, +-- which mxbuild refuses: CE6771 "It is not possible to create associations +-- to/from View Entities." +-- +-- Root cause of (3), and the whole fix: Studio Pro's association carries a +-- `Source` subdocument the plain one does not — +-- +-- "Source": { "$Type": "DomainModels$OqlViewAssociationSource", +-- "Reference": "persistent_order" } -- the OQL alias +-- +-- Measured on 11.13.0: adding exactly that takes the project from 2 errors +-- (CE6771 + CE6770) to 0. ONE field clears both, because the association is +-- also what makes the id column legal on the view entity. +-- +-- The fix derives the association from the OQL, as the platform does: the column +-- IS the declaration. That is also what makes describe → exec round-trip — +-- the OQL carries the column, so a described view entity rebuilds its +-- association with no second statement to keep in step. A plain +-- `create association` with a view entity at either end is now refused at +-- check AND exec time, pointing at this form. +-- +-- Measured, 11.13.0, this file: +-- pre-fix exec + mx check -> 2 × CE1613 (no association was created at all) +-- fixed exec + mx check -> 0 errors +-- re-run "Unchanged view entity", still 0 errors +-- describe -> exec into a fresh project -> 0 errors +-- MXCLI_ENGINE=legacy -> 0 errors +-- +-- Two refusals belong with this fix and are NOT in a .fail.mdl sibling: both +-- need a model to decide (is this endpoint a view entity? is that name taken in +-- the module?), and `make check-mdl` runs `check` with no project — naming such +-- a file .fail.mdl reports "negative test unexpectedly passed" and makes a +-- working rule look regressed (the Makefile's #891/#892 note). They are covered +-- by unit tests in mdl/executor/oql_view_associations_test.go; the measured +-- messages, with -p, are: +-- +-- create association VAssoc.BadRef from VAssoc.MeterTotalsVE to VAssoc.Meter; +-- -> cannot create association VAssoc.BadRef: VAssoc.MeterTotalsVE is a view +-- entity - Mendix does not allow a plain association to or from one +-- (CE6771). ... select t.ID as BadRef, ... +-- Refused at check AND exec time. The check existed before but SKIPPED an +-- endpoint the same script creates, which is the ordinary shape and so the +-- shape that got through; exec had no guard at all, so --no-check wrote it. +-- +-- ... select m.ID as meter, ... -- beside an entity called Meter +-- -> select alias 'meter' cannot be used: module 'VAssoc' already has an +-- entity of that name ... (case-insensitively). The alias IS the +-- association's name, so rename the alias - e.g. 'meterRef' +-- Mendix's own message is "Duplicate name 'Meter' in module 'VAssoc'. +-- Entities, associations and enumerations cannot share names." +-- ("found the hard way", FINDINGS §1.) + +create module VAssoc; + +create entity VAssoc.Meter ( + MeterCode: String(50), + Region: String(50) +); + +create entity VAssoc.Reading ( + Kwh: Decimal +); + +create association VAssoc.Reading_Meter from VAssoc.Reading to VAssoc.Meter; + +-- One attribute declared, TWO select columns. `m.ID as MeterRef` is the +-- association; `sum(r.Kwh) as TotalKwh` is the attribute. The association-path +-- join is how the target entity is reached, and the alias it binds (`m`) is what +-- makes `m.ID` resolvable. +create view entity VAssoc.MeterTotalsVE ( + TotalKwh: Decimal +) as ( + from VAssoc.Reading as r + join r/VAssoc.Reading_Meter/VAssoc.Meter as m + group by m.ID + select m.ID as MeterRef, sum(r.Kwh) as TotalKwh +); + +-- Control: the same id, taken as a STRING ATTRIBUTE instead. This is not an +-- association and must not become one — it is a different and often better +-- design (one SQL statement instead of two, no objects materialised in the +-- client), and the id is still there to look the real object up with. +create view entity VAssoc.MeterTotalsFlatVE ( + MeterId: String(200), + TotalKwh: Decimal +) as ( + from VAssoc.Reading as r + join r/VAssoc.Reading_Meter/VAssoc.Meter as m + group by m.ID + select cast(m.ID as string) as MeterId, sum(r.Kwh) as TotalKwh +); diff --git a/mdl/backend/modelsdk/association_move_write.go b/mdl/backend/modelsdk/association_move_write.go index 56c92a188..322a8d87e 100644 --- a/mdl/backend/modelsdk/association_move_write.go +++ b/mdl/backend/modelsdk/association_move_write.go @@ -62,6 +62,9 @@ func crossAssocToGen(ca *domainmodel.CrossModuleAssociation) *genDm.CrossAssocia } out.SetStorageFormat(sf) out.SetDeleteBehavior(deleteBehaviorToGen(behaviorType(ca.ParentDeleteBehavior), behaviorType(ca.ChildDeleteBehavior))) + if ca.Source == domainmodel.OqlViewAssociationSource { + out.SetSource(oqlViewAssociationSourceToGen(ca.ViewSourceReference)) + } return out } @@ -93,6 +96,13 @@ func crossAssocFromGenAssoc(a *genDm.Association, parentID, childRef string) *ge } } out.SetDeleteBehavior(deleteBehaviorToGen(pdb, cdb)) + // Moving the target entity to another module converts the association to a + // CrossAssociation. A view entity's Source has to survive that conversion, or + // the move — which never mentioned the association — silently breaks the + // build (CE6771). + if src, ok := a.Source().(*genDm.OqlViewAssociationSource); ok && src != nil { + out.SetSource(oqlViewAssociationSourceToGen(src.Reference())) + } return out } diff --git a/mdl/backend/modelsdk/domainmodel.go b/mdl/backend/modelsdk/domainmodel.go index cedef3750..62fb15356 100644 --- a/mdl/backend/modelsdk/domainmodel.go +++ b/mdl/backend/modelsdk/domainmodel.go @@ -602,6 +602,15 @@ func assocFromGen(a *genDm.Association) *domainmodel.Association { out.UpdatableFromParent = src.UpdatableFromParent() out.UpdatableFromChild = src.UpdatableFromChild() } + // A view entity's association to a persistent entity. Same reasoning as the + // OData source above and then some: an unread Source is written back as null, + // which is exactly the CE6771 + CE6770 pair the field exists to avoid — so a + // project that built yesterday stops building after any rewrite of its domain + // model, with no statement having mentioned the association. + if src, ok := a.Source().(*genDm.OqlViewAssociationSource); ok && src != nil { + out.Source = domainmodel.OqlViewAssociationSource + out.ViewSourceReference = src.Reference() + } return out } @@ -628,6 +637,10 @@ func crossAssocFromGen(ca *genDm.CrossAssociation) *domainmodel.CrossModuleAssoc ErrorMessage: deleteErrorMessageFromGen(db.ChildErrorMessage()), } } + if src, ok := ca.Source().(*genDm.OqlViewAssociationSource); ok && src != nil { + out.Source = domainmodel.OqlViewAssociationSource + out.ViewSourceReference = src.Reference() + } return out } diff --git a/mdl/backend/modelsdk/domainmodel_write.go b/mdl/backend/modelsdk/domainmodel_write.go index 0f2867123..09536b08b 100644 --- a/mdl/backend/modelsdk/domainmodel_write.go +++ b/mdl/backend/modelsdk/domainmodel_write.go @@ -799,10 +799,22 @@ func externalAssociationSourceToGen(a *domainmodel.Association) element.Element src := genRest.NewODataPrimitiveCollectionAssociationSource() assignID(src) return src + case domainmodel.OqlViewAssociationSource: + return oqlViewAssociationSourceToGen(a.ViewSourceReference) } return nil } +// oqlViewAssociationSourceToGen builds it. `Reference` is the OQL select alias; +// gen binds that exact storage name (checked against initOqlViewAssociationSource +// and against a real document), so no STORAGE-NAME OVERRIDE is needed here. +func oqlViewAssociationSourceToGen(reference string) element.Element { + src := genDm.NewOqlViewAssociationSource() + src.SetReference(reference) + assignID(src) + return src +} + // assignEntityIDs gives the entity, its generalization, and each attribute // (plus the attribute's type and stored value) fresh IDs (mirrors engalar's // assignEntityIDsGen). diff --git a/mdl/executor/cmd_associations.go b/mdl/executor/cmd_associations.go index 4b568eb8a..a5a637d9a 100644 --- a/mdl/executor/cmd_associations.go +++ b/mdl/executor/cmd_associations.go @@ -59,6 +59,19 @@ func execCreateAssociation(ctx *ExecContext, s *ast.CreateAssociationStmt) error } childID := childEntity.ID + // A view entity at either end is CE6771. `check` reports it too, but a script + // run with --no-check must not be able to write it — that is how the reported + // case got a model mxbuild refuses (FINDINGS §1), and it is the same #833 + // lesson as the module guard above. + for _, ep := range []struct { + entity *domainmodel.Entity + name string + }{{parentEntity, s.Parent.String()}, {childEntity, s.Child.String()}} { + if isViewEntity(ep.entity) { + return viewEntityAssociationRefusal(s.Name.String(), ep.name) + } + } + // Convert types assocType := domainmodel.AssociationTypeReference if s.Type == ast.AssocReferenceSet { diff --git a/mdl/executor/cmd_entities.go b/mdl/executor/cmd_entities.go index 142e4fb6c..6f61de286 100644 --- a/mdl/executor/cmd_entities.go +++ b/mdl/executor/cmd_entities.go @@ -848,6 +848,7 @@ func execCreateViewEntity(ctx *ExecContext, s *ast.CreateViewEntityStmt) error { OqlQuery: s.Query.RawQuery, } + created := entity if s.CreateOrModify && existingEntity != nil { // Update existing entity — preserve Source object ID to avoid CE-6770 entity.ID = existingEntity.ID @@ -873,6 +874,29 @@ func execCreateViewEntity(ctx *ExecContext, s *ast.CreateViewEntityStmt) error { fmt.Fprintf(ctx.Output, "Created view entity: %s\n", s.Name) } + // An `.ID` select column gives the view entity an ASSOCIATION to that + // entity, and Mendix has no separate declaration for it — the column is the + // declaration, so it is created here rather than by a statement of its own. + // Without this the OQL and the model disagree and mxbuild reports CE6770 + // "View Entity is out of sync with the OQL Query" (FINDINGS §1). + // + // After the write, so the entity has an ID to point the association's FROM + // end at, and re-read so a freshly created entity is found by name. + if fresh, err := ctx.Backend.GetDomainModel(module.ID); err == nil { + if stored := fresh.FindEntityByName(s.Name.Name); stored != nil { + created = stored + } + } + if nameErrors := validateViewAssociationNames(ctx, s.Name.Module, s.Name.Name, s.Query.RawQuery, nil); len(nameErrors) > 0 { + return mdlerrors.NewValidationf("view entity '%s':\n - %s", + s.Name.String(), strings.Join(nameErrors, "\n - ")) + } + if err := syncViewEntityAssociations(ctx, module.ID, s.Name.Module, created, s.Query.RawQuery); err != nil { + return err + } + invalidateHierarchy(ctx) + invalidateDomainModelsCache(ctx) + return nil } diff --git a/mdl/executor/oql_type_inference.go b/mdl/executor/oql_type_inference.go index 50db3e941..78fc62280 100644 --- a/mdl/executor/oql_type_inference.go +++ b/mdl/executor/oql_type_inference.go @@ -40,8 +40,10 @@ func inferOQLTypes(ctx *ExecContext, oqlQuery string, declaredAttrs []ast.ViewAt // Extract FROM clause and build alias map aliasMap := extractAliasMap(oqlQuery) - // Parse column expressions - columnExprs := parseSelectColumns(selectClause) + // An `.ID` column declares an ASSOCIATION, not an attribute, so it has + // no declared attribute to line up with. Dropping it here is what makes the + // remaining columns correspond one-to-one — see attributeSelectColumns. + columnExprs := attributeSelectColumns(oqlQuery, parseSelectColumns(selectClause)) if len(columnExprs) != len(declaredAttrs) { warnings = append(warnings, fmt.Sprintf( "OQL select has %d columns but %d attributes declared", @@ -78,17 +80,43 @@ func extractAliasMap(oql string) map[string]string { aliasMap := make(map[string]string) // Match FROM Entity AS alias or FROM Entity alias patterns - // Also handles JOIN clauses - fromPattern := regexp.MustCompile(`(?i)\b(?:from|join)\s+([A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*)\s+(?:as\s+)?([A-Za-z_][A-Za-z0-9_]*)`) + // Also handles JOIN clauses. + // + // Either half of the qualified name may be QUOTED, and that is not exotic: + // an OQL reserved word has to be quoted to survive MxBuild (CE0174), so + // `from Mappings."Order" as o` is the only way to write a source entity + // called Order. Matching bare names only left `o` unresolved — no type + // inference for its columns, and `o.ID` unrecognisable as an association + // column, which is exactly the reported example (FINDINGS §1). + fromPattern := regexp.MustCompile(`(?i)\b(?:from|join)\s+(` + oqlIdent + `\.` + oqlIdent + `)\s+(?:as\s+)?([A-Za-z_][A-Za-z0-9_]*)`) matches := fromPattern.FindAllStringSubmatch(oql, -1) for _, match := range matches { if len(match) >= 3 { - entityName := match[1] + entityName := unquoteQualifiedOQLName(match[1]) alias := match[2] aliasMap[alias] = entityName } } + // An ASSOCIATION-PATH join — `join r/Mod.Reading_Meter/Mod.Meter as m` — + // binds its alias to the entity at the END of the path. The pattern above + // cannot see it, because what follows the keyword is a path rather than a + // qualified name, so `m` resolved to nothing: no type inference for any of + // its columns, and `m.ID` unrecognisable as an association column. That join + // form is the ordinary way to reach a related entity in Mendix OQL. + pathPattern := regexp.MustCompile( + `(?i)\b(?:from|join)\s+[A-Za-z_]\w*(?:/` + oqlIdent + `\.` + oqlIdent + `)+\s+(?:as\s+)?([A-Za-z_]\w*)`) + lastEntity := regexp.MustCompile(`(` + oqlIdent + `\.` + oqlIdent + `)\s*$`) + for _, match := range pathPattern.FindAllStringSubmatch(oql, -1) { + alias := match[1] + // The path is everything between the keyword and the alias. + path := strings.TrimSuffix(strings.TrimSpace(match[0]), alias) + path = strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(path), "as")) + if seg := lastEntity.FindStringSubmatch(path); seg != nil { + aliasMap[alias] = unquoteQualifiedOQLName(seg[1]) + } + } + return aliasMap } @@ -107,6 +135,18 @@ const oqlIdent = `(?:"[^"\r\n]*"|` + "`[^`\r\n]*`" + `|\w+)` // an alias cannot disagree about what an alias looks like. var oqlAliasSuffixRe = regexp.MustCompile(`(?i)\s+as\s+(` + oqlIdent + `)\s*$`) +// unquoteQualifiedOQLName strips quoting from each half of a Module.Entity name, +// so `Mappings."Order"` and `Mappings.Order` resolve to the same entity. The +// STORED query keeps its quotes — MxBuild needs them — but the name the alias +// denotes is the bare one. +func unquoteQualifiedOQLName(qn string) string { + parts := strings.SplitN(qn, ".", 2) + if len(parts) != 2 { + return unquoteOQLIdent(qn) + } + return unquoteOQLIdent(parts[0]) + "." + unquoteOQLIdent(parts[1]) +} + // unquoteOQLIdent strips the quoting from an OQL identifier, so an alias can be // compared against a declared attribute name. The stored QUERY keeps its quotes // — they are what MxBuild needs — but the NAME the alias denotes is the bare @@ -142,7 +182,9 @@ func ValidateOQLTypes(oql string, attrs []ast.ViewAttribute) []linter.Violation return violations } - columnExprs := parseSelectColumns(selectClause) + // Same skip as inferOQLTypes: an association column has no declared + // attribute, and leaving it in shifts every attribute onto its neighbour. + columnExprs := attributeSelectColumns(oql, parseSelectColumns(selectClause)) for i, expr := range columnExprs { if i >= len(attrs) { diff --git a/mdl/executor/oql_view_associations.go b/mdl/executor/oql_view_associations.go new file mode 100644 index 000000000..96a8ec614 --- /dev/null +++ b/mdl/executor/oql_view_associations.go @@ -0,0 +1,422 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "regexp" + "strings" + + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// Selecting a persistent entity's id under an alias does not give a view entity +// an ATTRIBUTE. It gives it an ASSOCIATION to that entity, whose name is the +// alias: +// +// create view entity Sales.OrdersVE ( +// order_date: DateTime -- one attribute… +// ) as ( +// from Sales."Order" as o +// select o.ID as persistent_order -- …but two columns +// , o.OrderDate as order_date +// ); +// +// Studio Pro creates the association member when the column is added, and there +// is no separate declaration anywhere — the column IS the declaration. mxcli +// therefore derives it the same way, which is also what makes describe → exec +// round-trip: the OQL carries the column, so re-executing a describe rebuilds +// the association without a second statement to keep in step. +// +// Three consequences, each of which was a defect before +// (ako/view-entity-examples FINDINGS §1): +// +// - the id column has no declared attribute, so it must be skipped when +// columns are aligned with attributes — otherwise every attribute after it +// is checked against the wrong column; +// - the association must actually be created, or mxbuild reports CE6770 +// "View Entity is out of sync with the OQL Query"; +// - it must carry an OqlViewAssociationSource, or mxbuild reports CE6771 +// "It is not possible to create associations to/from View Entities". + +// viewAssociationColumn is one `.ID as ` select column. +type viewAssociationColumn struct { + // Name is the select alias. It becomes BOTH the association's name and the + // OqlViewAssociationSource's Reference — Studio Pro writes the same string + // in both places. + Name string + // Entity is the qualified name of the entity whose id is selected, resolved + // from the FROM/JOIN clause. Empty when the source alias could not be + // resolved, in which case this is not treated as an association column at + // all: guessing an entity would write a dangling reference. + Entity string + // Expr is the column as written, for diagnostics. + Expr string +} + +// oqlIDColumnRe matches a bare `.ID` select expression. Bare on purpose: +// `cast(m.ID as string) as MeterId` is a perfectly good STRING ATTRIBUTE +// carrying the object id as text — a different and often better design (one +// statement instead of two, no objects materialised in the client) — and it +// must not be mistaken for an association. +var oqlIDColumnRe = regexp.MustCompile(`(?i)^([A-Za-z_]\w*)\s*\.\s*id$`) + +// viewAssociationColumns returns the select columns of oql that declare an +// association, in select order. +func viewAssociationColumns(oql string) []viewAssociationColumn { + selectClause := extractSelectClause(oql) + if selectClause == "" { + return nil + } + aliases := extractAliasMap(oql) + var out []viewAssociationColumn + for _, expr := range parseSelectColumns(selectClause) { + col, ok := viewAssociationColumnOf(expr, aliases) + if ok { + out = append(out, col) + } + } + return out +} + +// viewAssociationColumnOf decides whether one select column declares an +// association, given the query's alias → entity map. +func viewAssociationColumnOf(expr string, aliases map[string]string) (viewAssociationColumn, bool) { + name := "" + if m := oqlAliasSuffixRe.FindStringSubmatch(expr); m != nil { + name = unquoteOQLIdent(m[1]) + expr = strings.TrimSpace(strings.TrimSuffix(expr, m[0])) + } + // No alias means no association: the alias is the association's name, and + // MDL030 already reports a select column without one. + if name == "" { + return viewAssociationColumn{}, false + } + m := oqlIDColumnRe.FindStringSubmatch(strings.TrimSpace(expr)) + if m == nil { + return viewAssociationColumn{}, false + } + entity := aliases[m[1]] + if entity == "" { + return viewAssociationColumn{}, false + } + return viewAssociationColumn{Name: name, Entity: entity, Expr: expr}, true +} + +// isViewAssociationColumn reports whether one select column (as written, alias +// included) declares an association rather than an attribute. +func isViewAssociationColumn(expr string, aliases map[string]string) bool { + _, ok := viewAssociationColumnOf(expr, aliases) + return ok +} + +// attributeSelectColumns drops the association columns from a select list, so +// what remains lines up ONE-TO-ONE with the declared attributes. +// +// This is the whole of the reported symptom 1. The alignment is positional, so +// an id column does not merely add an unchecked entry — it SHIFTS every column +// after it, and each remaining attribute is then reported against its +// neighbour's expression. With the id first, three correct attributes produced +// "attribute 'TotalKwh': declared as Decimal but OQL expression returns +// Integer"; with it last, "OQL select has 4 columns but 3 attributes declared". +// Neither describes the script. +func attributeSelectColumns(oql string, columns []string) []string { + aliases := extractAliasMap(oql) + out := make([]string, 0, len(columns)) + for _, c := range columns { + if isViewAssociationColumn(c, aliases) { + continue + } + out = append(out, c) + } + return out +} + +// syncViewEntityAssociations makes the view entity's associations match its OQL: +// one association per `.ID as ` column, and none left over from a +// column that has since been removed. +// +// It runs as part of CREATE VIEW ENTITY rather than being a statement of its +// own, because in Mendix the column IS the declaration — there is nothing else +// for a separate statement to say, and a second statement could disagree with +// the OQL, which is precisely the state (CE6770) this exists to prevent. +// +// Only associations mxcli can prove it owns are removed: one carrying an +// OqlViewAssociationSource, whose FROM end is this view entity. A hand-written +// association between other entities is never touched. +func syncViewEntityAssociations(ctx *ExecContext, moduleID model.ID, moduleName string, + viewEntity *domainmodel.Entity, oql string) error { + + dm, err := ctx.Backend.GetDomainModel(moduleID) + if err != nil { + return mdlerrors.NewBackend("get domain model", err) + } + + wanted := viewAssociationColumns(oql) + keep := make(map[string]bool, len(wanted)) + for _, c := range wanted { + keep[c.Name] = true + } + + // Drop the ones this view entity used to have and no longer declares. + var dropped []string + for _, a := range dm.Associations { + if a.Source == domainmodel.OqlViewAssociationSource && a.ParentID == viewEntity.ID && !keep[a.Name] { + if err := ctx.Backend.DeleteAssociation(dm.ID, a.ID); err != nil { + return mdlerrors.NewBackend("remove stale view association", err) + } + dropped = append(dropped, a.Name) + } + } + for _, ca := range dm.CrossAssociations { + if ca.Source == domainmodel.OqlViewAssociationSource && ca.ParentID == viewEntity.ID && !keep[ca.Name] { + if err := ctx.Backend.DeleteCrossAssociation(dm.ID, ca.ID); err != nil { + return mdlerrors.NewBackend("remove stale view association", err) + } + dropped = append(dropped, ca.Name) + } + } + for _, name := range dropped { + fmt.Fprintf(ctx.Output, "Removed view association: %s.%s (its OQL column is gone)\n", moduleName, name) + } + + for _, col := range wanted { + if err := upsertViewAssociation(ctx, moduleID, moduleName, viewEntity, col); err != nil { + return err + } + } + return nil +} + +// upsertViewAssociation creates or updates the one association a column declares. +func upsertViewAssociation(ctx *ExecContext, moduleID model.ID, moduleName string, + viewEntity *domainmodel.Entity, col viewAssociationColumn) error { + + parts := strings.Split(col.Entity, ".") + if len(parts) != 2 { + return mdlerrors.NewValidationf( + "select column '%s as %s' names an association to %q, which is not a qualified entity name", + col.Expr, col.Name, col.Entity) + } + targetModule, targetName := parts[0], parts[1] + target, err := findEntity(ctx, targetModule, targetName) + if err != nil { + return mdlerrors.NewNotFoundMsg("entity", col.Entity, fmt.Sprintf( + "select column '%s as %s' gives the view entity an association to %s, which does not exist", + col.Expr, col.Name, col.Entity)) + } + + dm, err := ctx.Backend.GetDomainModel(moduleID) + if err != nil { + return mdlerrors.NewBackend("get domain model", err) + } + + // An association is stored in the module of its FROM entity, which here is + // always the view entity — so this domain model is always the right one, and + // cross-module only ever refers to where the TARGET lives. + crossModule := targetModule != moduleName + + for _, a := range dm.Associations { + if a.Name != col.Name { + continue + } + if a.ParentID != viewEntity.ID { + return viewAssociationNameClash(moduleName, col.Name) + } + if crossModule { + // The target moved to another module: the stored form has to change + // type, which is a delete + create rather than an update. + if err := ctx.Backend.DeleteAssociation(dm.ID, a.ID); err != nil { + return mdlerrors.NewBackend("replace view association", err) + } + break + } + a.ChildID = target.ID + a.Source = domainmodel.OqlViewAssociationSource + a.ViewSourceReference = col.Name + if err := ctx.Backend.UpdateDomainModel(dm); err != nil { + return mdlerrors.NewBackend("update view association", err) + } + return nil + } + for _, ca := range dm.CrossAssociations { + if ca.Name != col.Name { + continue + } + if ca.ParentID != viewEntity.ID { + return viewAssociationNameClash(moduleName, col.Name) + } + if !crossModule { + if err := ctx.Backend.DeleteCrossAssociation(dm.ID, ca.ID); err != nil { + return mdlerrors.NewBackend("replace view association", err) + } + break + } + ca.ChildRef = col.Entity + ca.Source = domainmodel.OqlViewAssociationSource + ca.ViewSourceReference = col.Name + if err := ctx.Backend.UpdateDomainModel(dm); err != nil { + return mdlerrors.NewBackend("update view association", err) + } + return nil + } + + keep := &domainmodel.DeleteBehavior{Type: domainmodel.DeleteBehaviorTypeDeleteMeButKeepReferences} + if crossModule { + ca := &domainmodel.CrossModuleAssociation{ + Name: col.Name, + Type: domainmodel.AssociationTypeReference, + Owner: domainmodel.AssociationOwnerDefault, + StorageFormat: domainmodel.StorageFormatColumn, + ParentID: viewEntity.ID, + ChildRef: col.Entity, + ChildDeleteBehavior: keep, + Source: domainmodel.OqlViewAssociationSource, + ViewSourceReference: col.Name, + } + if err := ctx.Backend.CreateCrossAssociation(dm.ID, ca); err != nil { + return mdlerrors.NewBackend("create view association", err) + } + } else { + a := &domainmodel.Association{ + Name: col.Name, + Type: domainmodel.AssociationTypeReference, + Owner: domainmodel.AssociationOwnerDefault, + StorageFormat: domainmodel.StorageFormatColumn, + ParentID: viewEntity.ID, + ChildID: target.ID, + ChildDeleteBehavior: keep, + Source: domainmodel.OqlViewAssociationSource, + ViewSourceReference: col.Name, + } + if err := ctx.Backend.CreateAssociation(dm.ID, a); err != nil { + return mdlerrors.NewBackend("create view association", err) + } + } + fmt.Fprintf(ctx.Output, "Created view association: %s.%s -> %s\n", moduleName, col.Name, col.Entity) + return nil +} + +// viewAssociationNameClash reports an alias colliding with something else in the +// module. Mendix's own message is "Duplicate name '' in module ''. +// Entities, associations and enumerations cannot share names." — and it is +// CASE-INSENSITIVE, so `as meter` beside an entity called `Meter` collides +// (ako/view-entity-examples FINDINGS §1). +func viewAssociationNameClash(moduleName, name string) error { + return mdlerrors.NewValidationf( + "select alias '%s' cannot name this view entity's association: module '%s' already has a "+ + "different association of that name.\n Rename the OQL alias — the alias IS the "+ + "association's name, and Mendix requires it to be unique among the module's entities, "+ + "associations and enumerations (case-insensitively)", + name, moduleName) +} + +// viewEntityAssociationRefusal is the CE6771 message. Mendix rejects an +// association with a view entity at either end, and there is nothing to write +// instead — a plain association there is the thing the platform refuses. +// +// The advice used to be "use a non-persistent entity with a real reference to +// the target instead", which was a workaround for a capability mxcli did not +// have. It has it now: selecting the target's id under an alias gives the view +// entity exactly this association, built the way Studio Pro builds it. Pointing +// at that is the difference between refusing the statement and refusing the +// goal. +func viewEntityAssociationRefusal(assocName, endpoint string) error { + return mdlerrors.NewValidationf( + "cannot create association %s: %s is a view entity — Mendix does not allow a plain "+ + "association to or from one (CE6771).\n"+ + " A view entity gets an association by SELECTING THE TARGET'S ID under an alias, "+ + "which is also the association's name:\n"+ + " select t.ID as %s, … -- t is the target entity's alias in the FROM/JOIN\n"+ + " mxcli creates the association from that column, with the "+ + "OqlViewAssociationSource the platform requires. There is no separate statement for it, "+ + "because in Mendix the column IS the declaration", + assocName, endpoint, shortAssociationName(assocName)) +} + +// shortAssociationName is the bare name of a possibly-qualified association, for +// use inside an example. +func shortAssociationName(qualified string) string { + if i := strings.LastIndex(qualified, "."); i >= 0 { + return qualified[i+1:] + } + return qualified +} + +// validateViewAssociationNames reports select aliases that cannot become +// association names. +// +// The alias is the association's name, and Mendix requires that name to be +// unique among the module's entities, associations and enumerations — +// CASE-INSENSITIVELY. `select m.ID as meter` beside an entity called `Meter` +// fails with "Duplicate name 'Meter' in module 'Trends'. Entities, associations +// and enumerations cannot share names." (FINDINGS §1, "found the hard way"). +// +// Reported here rather than left to mxbuild because the fix is a rename, and a +// rename is much cheaper before the name has spread through pages and +// microflows. Shared by `check` and `exec` so the two cannot disagree. +func validateViewAssociationNames(ctx *ExecContext, moduleName, viewEntityName, oql string, + scriptEntities map[string]bool) []string { + + cols := viewAssociationColumns(oql) + if len(cols) == 0 { + return nil + } + taken := map[string]string{} // lower-case name -> what holds it + + if dms, err := ctx.Backend.ListDomainModels(); err == nil { + if h, herr := getHierarchy(ctx); herr == nil { + for _, dm := range dms { + if h.GetModuleName(dm.ContainerID) != moduleName { + continue + } + for _, e := range dm.Entities { + if e.Name != viewEntityName { + taken[strings.ToLower(e.Name)] = "an entity" + } + } + for _, a := range dm.Associations { + if a.Source != domainmodel.OqlViewAssociationSource { + taken[strings.ToLower(a.Name)] = "an association" + } + } + for _, ca := range dm.CrossAssociations { + if ca.Source != domainmodel.OqlViewAssociationSource { + taken[strings.ToLower(ca.Name)] = "an association" + } + } + } + } + } + // Entities the same script creates are not in the project yet. + for qn := range scriptEntities { + parts := strings.SplitN(qn, ".", 2) + if len(parts) == 2 && parts[0] == moduleName && parts[1] != viewEntityName { + taken[strings.ToLower(parts[1])] = "an entity" + } + } + + var out []string + seen := map[string]bool{} + for _, c := range cols { + lower := strings.ToLower(c.Name) + if what, clash := taken[lower]; clash { + out = append(out, fmt.Sprintf( + "select alias '%s' cannot be used: module '%s' already has %s of that name, and "+ + "Mendix reports \"Duplicate name\" for entities, associations and enumerations "+ + "sharing one (case-insensitively). The alias IS the association's name, so rename "+ + "the alias — e.g. '%sRef'", + c.Name, moduleName, what, c.Name)) + continue + } + if seen[lower] { + out = append(out, fmt.Sprintf( + "select alias '%s' is used by two association columns — each one names a separate "+ + "association, so the names have to differ", c.Name)) + } + seen[lower] = true + } + return out +} diff --git a/mdl/executor/oql_view_associations_test.go b/mdl/executor/oql_view_associations_test.go new file mode 100644 index 000000000..68342a1fd --- /dev/null +++ b/mdl/executor/oql_view_associations_test.go @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// A view entity's association is declared by an `.ID` select column and +// by nothing else. These tests cover the recognition half — which column is an +// association, which entity it points at, and what that does to the alignment +// between columns and declared attributes. The writing half (the +// OqlViewAssociationSource, without which mxbuild reports CE6771) is measured +// end to end; see mdl-examples/bug-tests/view-entity-association.mdl. + +func TestViewAssociationColumns_RecognisesTheIDColumn(t *testing.T) { + cols := viewAssociationColumns( + `select o.ID as persistent_order, o.OrderDate as order_date from Sales."Order" as o`) + if len(cols) != 1 { + t.Fatalf("got %d association columns, want 1: %+v", len(cols), cols) + } + if cols[0].Name != "persistent_order" { + t.Errorf("Name = %q, want persistent_order — the select alias IS the association name", cols[0].Name) + } + // The quoted source entity is the reported example: `Order` is an OQL + // reserved word, so it can only be written quoted, and the alias map has to + // see through the quotes or the column is not recognised at all. + if cols[0].Entity != "Sales.Order" { + t.Errorf("Entity = %q, want Sales.Order", cols[0].Entity) + } +} + +func TestViewAssociationColumns_ResolvesAnAssociationPathJoin(t *testing.T) { + // The ordinary way to reach a related entity in Mendix OQL. The alias map + // used to match only `join Module.Entity as x`, so `m` resolved to nothing + // and `m.ID` was invisible. + cols := viewAssociationColumns( + `from Trends.Reading as r join r/Trends.Reading_Meter/Trends.Meter as m ` + + `group by m.ID select m.ID as MeterRef, sum(r.Kwh) as TotalKwh`) + if len(cols) != 1 || cols[0].Entity != "Trends.Meter" || cols[0].Name != "MeterRef" { + t.Fatalf("got %+v, want one column MeterRef -> Trends.Meter", cols) + } +} + +// TestViewAssociationColumns_LeavesEverythingElseAlone is the control, and it +// carries the design decision. `cast(m.ID as string) as MeterId` is a STRING +// ATTRIBUTE holding the object id as text — a legitimate and often better +// design (one statement instead of two, no objects materialised in the client), +// and treating it as an association would write a member the author did not ask +// for. +func TestViewAssociationColumns_LeavesEverythingElseAlone(t *testing.T) { + for _, oql := range []string{ + `select cast(m.ID as string) as MeterId from Trends.Meter as m`, + `select m.MeterCode as MeterCode from Trends.Meter as m`, + `select sum(r.Kwh) as TotalKwh from Trends.Reading as r`, + // An unresolvable alias: guessing an entity would write a dangling + // reference, so this is not an association column. + `select x.ID as Something from Trends.Meter as m`, + // No alias at all — MDL030 reports that; there is no name to use. + `select m.ID from Trends.Meter as m`, + } { + if cols := viewAssociationColumns(oql); len(cols) != 0 { + t.Errorf("%s\n -> unexpectedly read as an association: %+v", oql, cols) + } + } +} + +// TestIDColumnDoesNotShiftTheAttributeAlignment is reported symptom 1. Columns +// are matched to declared attributes BY POSITION, so an unrecognised id column +// does not merely go unchecked — it moves every attribute after it onto its +// neighbour's expression, and the resulting message describes a script nobody +// wrote. +func TestIDColumnDoesNotShiftTheAttributeAlignment(t *testing.T) { + attrs := []ast.ViewAttribute{ + {Name: "MeterCode", Type: ast.DataType{Kind: ast.TypeString, Length: 200}}, + {Name: "Readings", Type: ast.DataType{Kind: ast.TypeInteger}}, + {Name: "TotalKwh", Type: ast.DataType{Kind: ast.TypeDecimal}}, + } + const body = `cast(m.Code as string) as MeterCode, count(r.Kwh) as Readings, avg(r.Kwh) as TotalKwh` + const from = ` from Trends.Reading as r join r/Trends.R_M/Trends.Meter as m` + + for _, c := range []struct{ name, oql string }{ + {"id first", `select m.ID as MeterRef, ` + body + from}, + {"id last", `select ` + body + `, m.ID as MeterRef` + from}, + {"no id column (control)", `select ` + body + from}, + } { + if v := ValidateOQLTypes(c.oql, attrs); len(v) != 0 { + t.Errorf("%s: %d spurious violation(s): %s", c.name, len(v), v[0].Message) + } + } + + // And the count check agrees: three attributes, three attribute columns, + // however many association columns sit among them. + cols := attributeSelectColumns( + `select m.ID as MeterRef, `+body+from, + parseSelectColumns(extractSelectClause(`select m.ID as MeterRef, `+body+from))) + if len(cols) != 3 { + t.Errorf("attribute columns = %d, want 3: %q", len(cols), cols) + } +} + +// TestIDColumnStillNeedsAnAlias: the alias is the association's name, so the +// MDL030 "no as alias" rule has to keep applying to an id column. Skipping +// association columns from the TYPE alignment must not skip them from the +// syntax rules. +func TestIDColumnStillNeedsAnAlias(t *testing.T) { + ids := oqlRuleIDs(`select m.ID, m.MeterCode as MeterCode from Trends.Meter as m`) + if !hasOQLRule(ids, "MDL030") { + t.Errorf("an id column with no alias was accepted: %v", ids) + } +} + +func TestViewEntityAssociationRefusal_PointsAtTheColumnForm(t *testing.T) { + err := viewEntityAssociationRefusal("Trends.MeterRef", "Trends.MeterTotalsVE") + msg := err.Error() + for _, want := range []string{"CE6771", "select t.ID as MeterRef"} { + if !strings.Contains(msg, want) { + t.Errorf("refusal does not mention %q:\n%s", want, msg) + } + } + // The old advice was a workaround for a capability mxcli now has; it must + // not survive, or the message sends the author away from the real answer. + if strings.Contains(msg, "non-persistent entity") { + t.Errorf("refusal still recommends the pre-fix workaround:\n%s", msg) + } +} + +func TestShortAssociationName(t *testing.T) { + if got := shortAssociationName("Mod.Assoc"); got != "Assoc" { + t.Errorf("got %q, want Assoc", got) + } + if got := shortAssociationName("Assoc"); got != "Assoc" { + t.Errorf("got %q, want Assoc", got) + } +} diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 6bccfff1c..a5d4a065d 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -19,8 +19,14 @@ import ( // scriptContext holds objects defined within a script for reference validation. type scriptContext struct { - modules map[string]bool // Modules created in the script - entities map[string]bool // Entities created (Module.Entity) + modules map[string]bool // Modules created in the script + entities map[string]bool // Entities created (Module.Entity) + // viewEntities is the subset of entities that are VIEW entities. Kept apart + // because a view entity is refused where a persistent one is fine (CE6771), + // and the endpoint check below skips anything the script creates — so + // without this, creating the view entity and the association in one script + // (the ordinary shape) walked straight past the rule. + viewEntities map[string]bool enumerations map[string]bool // Enumerations created (Module.Enum) microflows map[string]bool // Microflows created (Module.Microflow) nanoflows map[string]bool // Nanoflows created (Module.Nanoflow) @@ -63,6 +69,7 @@ func newScriptContext() *scriptContext { return &scriptContext{ modules: make(map[string]bool), entities: make(map[string]bool), + viewEntities: make(map[string]bool), enumerations: make(map[string]bool), microflows: make(map[string]bool), nanoflows: make(map[string]bool), @@ -123,69 +130,17 @@ func codeActionParamNames(params []ast.JavaActionParam) []string { } // collectDefinitions scans a program and collects all objects that will be created. +// collectDefinitions records every object a program defines. +// +// It is a loop over collectSingle, and deliberately nothing more. The two used +// to be parallel switch statements over the same statement types, kept in step +// by hand — and they were not in step: collectSingle had no CreateConstantStmt +// case, and adding view-entity tracking to one of them left the other silent, +// so an association to a view entity created by the SAME script walked past the +// CE6771 rule. One list beats two agreeing lists. func (sc *scriptContext) collectDefinitions(prog *ast.Program) { for _, stmt := range prog.Statements { - switch s := stmt.(type) { - case *ast.CreateModuleStmt: - sc.modules[s.Name] = true - case *ast.CreateEntityStmt: - if s.Name.Module != "" { - sc.entities[s.Name.String()] = true - sc.recordEntityAttrs(s) - } - case *ast.CreateAssociationStmt: - sc.recordAssociation(s) - case *ast.CreateViewEntityStmt: - if s.Name.Module != "" { - sc.entities[s.Name.String()] = true - } - case *ast.CreateExternalEntityStmt: - if s.Name.Module != "" { - sc.entities[s.Name.String()] = true - } - case *ast.CreateEnumerationStmt: - if s.Name.Module != "" { - sc.enumerations[s.Name.String()] = true - } - case *ast.CreateConstantStmt: - if s.Name.Module != "" { - sc.constants[s.Name.String()] = true - } - case *ast.CreateMicroflowStmt: - if s.Name.Module != "" { - sc.microflows[s.Name.String()] = true - sc.recordFlowParams(s.Name.String(), s.Parameters, s.ReturnType) - } - case *ast.CreateNanoflowStmt: - if s.Name.Module != "" { - sc.nanoflows[s.Name.String()] = true - sc.recordFlowParams(s.Name.String(), s.Parameters, s.ReturnType) - } - case *ast.CreatePageStmtV3: - if s.Name.Module != "" { - sc.pages[s.Name.String()] = true - } - case *ast.CreateSnippetStmtV3: - if s.Name.Module != "" { - sc.snippets[s.Name.String()] = true - } - case *ast.CreateLayoutStmt: - if s.Name.Module != "" { - sc.layouts[s.Name.String()] = true - } - case *ast.CreateWorkflowStmt: - if s.Name.Module != "" { - sc.workflows[s.Name.String()] = true - } - case *ast.CreateJavaActionStmt: - if s.Name.Module != "" { - sc.javaActions[s.Name.String()] = codeActionParamNames(s.Parameters) - } - case *ast.CreateJavaScriptActionStmt: - if s.Name.Module != "" { - sc.javaScriptActions[s.Name.String()] = codeActionParamNames(s.Parameters) - } - } + sc.collectSingle(stmt) } } @@ -204,6 +159,7 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { case *ast.CreateViewEntityStmt: if s.Name.Module != "" { sc.entities[s.Name.String()] = true + sc.viewEntities[s.Name.String()] = true } case *ast.CreateExternalEntityStmt: if s.Name.Module != "" { @@ -213,6 +169,10 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { if s.Name.Module != "" { sc.enumerations[s.Name.String()] = true } + case *ast.CreateConstantStmt: + if s.Name.Module != "" { + sc.constants[s.Name.String()] = true + } case *ast.CreateMicroflowStmt: if s.Name.Module != "" { sc.microflows[s.Name.String()] = true @@ -539,13 +499,17 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext // the same script are skipped (a view entity created here is validated on its // own statement). (ledger finding #41) for _, ep := range []ast.QualifiedName{s.Parent, s.Child} { - if ep.Module == "" || sc.entities[ep.String()] { + if ep.Module == "" { + continue + } + if sc.viewEntities[ep.String()] { + return viewEntityAssociationRefusal(s.Name.String(), ep.String()) + } + if sc.entities[ep.String()] { continue } if ent, err := findEntity(ctx, ep.Module, ep.Name); err == nil && isViewEntity(ent) { - return mdlerrors.NewValidationf( - "cannot create association %s: %s is a view entity — Mendix does not allow associations to or from view entities (CE6771). Use a non-persistent entity with a real reference to the target instead.", - s.Name.String(), ep.String()) + return viewEntityAssociationRefusal(s.Name.String(), ep.String()) } } case *ast.CreateImageCollectionStmt: @@ -690,6 +654,13 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext return mdlerrors.NewNotFound("module", s.Name.Module) } } + // An `.ID` column names an association, and the name has to be + // free in the module — reported here because the fix is a rename, and a + // rename is cheapest before the name spreads (FINDINGS §1). + if nameErrors := validateViewAssociationNames(ctx, s.Name.Module, s.Name.Name, s.Query.RawQuery, sc.entities); len(nameErrors) > 0 { + return mdlerrors.NewValidationf("view entity '%s':\n - %s", + s.Name.String(), strings.Join(nameErrors, "\n - ")) + } // Validate OQL types match declared attribute types if typeErrors := validateViewEntityTypes(ctx, s); len(typeErrors) > 0 { return mdlerrors.NewValidationf("view entity '%s' has type mismatches:\n - %s", diff --git a/sdk/domainmodel/domainmodel.go b/sdk/domainmodel/domainmodel.go index 5f5f4ae58..f382fcf4c 100644 --- a/sdk/domainmodel/domainmodel.go +++ b/sdk/domainmodel/domainmodel.go @@ -354,7 +354,25 @@ type Association struct { // external entities). When Source = "Rest$ODataRemoteAssociationSource", // the writer emits a Source block carrying the OData navigation property // names instead of leaving the association as a plain persistent one. - Source string `json:"source,omitempty"` + // + // A VIEW ENTITY's association to a persistent entity uses the same slot, + // with Source = "DomainModels$OqlViewAssociationSource" and + // ViewSourceReference naming the OQL select alias it comes from. Measured on + // 11.13.0: that one subdocument is the whole difference between a model + // mxbuild accepts and one it refuses with CE6771 "It is not possible to + // create associations to/from View Entities" — and it also clears the + // CE6770 the id column otherwise causes on the view entity itself. + Source string `json:"source,omitempty"` + + // ViewSourceReference is the OQL select alias an OqlViewAssociationSource + // points at. It equals the association's own Name in everything Studio Pro + // writes, but it is stored separately and read back separately: they are two + // different things (a name and a column reference), and treating them as one + // would silently rename the reference whenever the association is renamed. + ViewSourceReference string `json:"viewSourceReference,omitempty"` + + // (see OqlViewAssociationSource for the $Type this Source takes) + RemoteParentNavigationProperty string `json:"remoteParentNavigationProperty,omitempty"` RemoteChildNavigationProperty string `json:"remoteChildNavigationProperty,omitempty"` CreatableFromParent bool `json:"creatableFromParent,omitempty"` @@ -364,6 +382,24 @@ type Association struct { Navigability2 string `json:"navigability2,omitempty"` // "ParentToChild" or "BothDirections" } +// OqlViewAssociationSource is the Source $Type that makes an association to a +// VIEW ENTITY legal, and the one thing that distinguishes it from an ordinary +// persistent association. +// +// Measured on Mendix 11.13.0: adding exactly this subdocument — three keys, +// $ID / $Type / Reference — takes a project from +// +// [CE6771] "It is not possible to create associations to/from View Entities." +// [CE6770] "View Entity is out of sync with the OQL Query." +// +// to 0 errors. One field clears both, because the association is also what makes +// the OQL's `.ID` select column legal on the view entity. +// +// It lives here, in the semantic model, rather than in either backend: both +// engines write it and both read it, and a second copy of the string is how the +// two would come to disagree. +const OqlViewAssociationSource = "DomainModels$OqlViewAssociationSource" + // GetName returns the association's name. func (a *Association) GetName() string { return a.Name @@ -623,6 +659,15 @@ type CrossModuleAssociation struct { StorageFormat AssociationStorageFormat `json:"storageFormat,omitempty"` ParentDeleteBehavior *DeleteBehavior `json:"parentDeleteBehavior,omitempty"` ChildDeleteBehavior *DeleteBehavior `json:"childDeleteBehavior,omitempty"` + + // Source / ViewSourceReference carry a view entity's OqlViewAssociationSource, + // exactly as on Association. A view entity's association is cross-module + // whenever the entity it points at lives elsewhere, which is the shape + // ako/view-entity-examples reported it in — so the two association types have + // to grow the field together or the fix works for one project layout and not + // the other. + Source string `json:"source,omitempty"` + ViewSourceReference string `json:"viewSourceReference,omitempty"` } // GetName returns the cross-module association's name. diff --git a/sdk/mpr/parser_domainmodel.go b/sdk/mpr/parser_domainmodel.go index 1a6593705..26a22df45 100644 --- a/sdk/mpr/parser_domainmodel.go +++ b/sdk/mpr/parser_domainmodel.go @@ -475,6 +475,13 @@ func parseAssociation(raw map[string]any) *domainmodel.Association { assoc.Navigability2 = extractString(sourceMap["Navigability2"]) case "Rest$ODataPrimitiveCollectionAssociationSource": assoc.Source = "Rest$ODataPrimitiveCollectionAssociationSource" + case "DomainModels$OqlViewAssociationSource": + // A view entity's association to a persistent entity. Reading it is + // not a convenience: an unread Source is written back as null on the + // next rewrite of this domain model, which turns a working project + // into CE6771 + CE6770 with no statement having asked for that. + assoc.Source = "DomainModels$OqlViewAssociationSource" + assoc.ViewSourceReference = extractString(sourceMap["Reference"]) } } @@ -513,6 +520,15 @@ func parseCrossAssociation(raw map[string]any) *domainmodel.CrossModuleAssociati } } + // A view entity pointing at an entity in ANOTHER module lands here rather + // than in parseAssociation, so the Source has to be read in both places. + if sourceMap, ok := raw["Source"].(map[string]any); ok { + if extractString(sourceMap["$Type"]) == "DomainModels$OqlViewAssociationSource" { + ca.Source = "DomainModels$OqlViewAssociationSource" + ca.ViewSourceReference = extractString(sourceMap["Reference"]) + } + } + return ca } diff --git a/sdk/mpr/writer_domainmodel.go b/sdk/mpr/writer_domainmodel.go index 5874a8adb..8801abf66 100644 --- a/sdk/mpr/writer_domainmodel.go +++ b/sdk/mpr/writer_domainmodel.go @@ -1207,6 +1207,8 @@ func serializeAssociation(a *domainmodel.Association) bson.D { {Key: "$ID", Value: idToBsonBinary(generateUUID())}, {Key: "$Type", Value: "Rest$ODataPrimitiveCollectionAssociationSource"}, } + case domainmodel.OqlViewAssociationSource: + source = oqlViewAssociationSourceDoc(a.ViewSourceReference) default: source = nil } @@ -1250,11 +1252,32 @@ func serializeCrossAssociation(ca *domainmodel.CrossModuleAssociation) bson.D { {Key: "Type", Value: string(ca.Type)}, {Key: "Owner", Value: string(ca.Owner)}, {Key: "StorageFormat", Value: storageFormat}, - {Key: "Source", Value: nil}, + {Key: "Source", Value: crossAssociationSource(ca)}, {Key: "DeleteBehavior", Value: serializeDeleteBehavior(ca.ParentDeleteBehavior, ca.ChildDeleteBehavior)}, } } +// oqlViewAssociationSourceDoc builds that subdocument. Three keys and no more — +// the shape is pinned against a Studio Pro document (ako/TestApp, 11.14) and +// re-measured here on 11.13.0. +func oqlViewAssociationSourceDoc(reference string) bson.D { + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(generateUUID())}, + {Key: "$Type", Value: domainmodel.OqlViewAssociationSource}, + {Key: "Reference", Value: reference}, + } +} + +func crossAssociationSource(ca *domainmodel.CrossModuleAssociation) any { + if ca.Source == domainmodel.OqlViewAssociationSource { + return oqlViewAssociationSourceDoc(ca.ViewSourceReference) + } + // A CrossAssociation has never carried an OData source — those live between + // external entities, which are not cross-module — so nil stays the default + // rather than being widened speculatively. + return nil +} + func serializeDeleteBehavior(parentBehavior, childBehavior *domainmodel.DeleteBehavior) bson.D { parentType := "DeleteMeButKeepReferences" childType := "DeleteMeButKeepReferences" diff --git a/sdk/mpr/writer_domainmodel_test.go b/sdk/mpr/writer_domainmodel_test.go index 2ddad30d7..dac37dbba 100644 --- a/sdk/mpr/writer_domainmodel_test.go +++ b/sdk/mpr/writer_domainmodel_test.go @@ -244,3 +244,66 @@ func TestSerializeAssociation_PreservesConnectionPoints(t *testing.T) { t.Errorf("ChildConnection = %v, want \"0;0\" — the zero point is a value, not an absence", got["ChildConnection"]) } } + +// TestSerializeAssociation_OqlViewSource pins the one field that makes an +// association to a VIEW ENTITY legal. Measured on Mendix 11.13.0: without it +// mxbuild reports CE6771 "It is not possible to create associations to/from +// View Entities" AND CE6770 on the view entity; adding exactly this three-key +// subdocument takes the same project to 0 errors. +func TestSerializeAssociation_OqlViewSource(t *testing.T) { + a := &domainmodel.Association{ + Name: "MeterRef", + Type: domainmodel.AssociationTypeReference, + Owner: domainmodel.AssociationOwnerDefault, + StorageFormat: domainmodel.StorageFormatColumn, + Source: domainmodel.OqlViewAssociationSource, + ViewSourceReference: "MeterRef", + } + got := dToM(serializeAssociation(a)) + m, ok := got["Source"].(bson.M) + if !ok { + t.Fatalf("Source is %T, want a subdocument", got["Source"]) + } + if m["$Type"] != domainmodel.OqlViewAssociationSource { + t.Errorf("$Type = %v", m["$Type"]) + } + if m["Reference"] != "MeterRef" { + t.Errorf("Reference = %v, want the OQL select alias", m["Reference"]) + } + // Three keys and no more — the shape is pinned against a Studio Pro document. + if len(m) != 3 { + t.Errorf("Source has %d keys, want exactly $ID/$Type/Reference: %v", len(m), m) + } + + // Control: an ordinary association still writes a null Source. Widening the + // switch must not start decorating every association. + plain := dToM(serializeAssociation(&domainmodel.Association{Name: "Plain"})) + if plain["Source"] != nil { + t.Errorf("a plain association got Source = %v, want nil", plain["Source"]) + } +} + +// A view entity pointing at an entity in another module is stored as a +// CrossAssociation, which is the shape the defect was reported in — so the two +// serializers have to carry the field together. +func TestSerializeCrossAssociation_OqlViewSource(t *testing.T) { + ca := &domainmodel.CrossModuleAssociation{ + Name: "persistent_order", + ChildRef: "Mappings.Order", + Source: domainmodel.OqlViewAssociationSource, + ViewSourceReference: "persistent_order", + } + m, ok := dToM(serializeCrossAssociation(ca))["Source"].(bson.M) + if !ok { + t.Fatalf("Source is %T, want a subdocument", dToM(serializeCrossAssociation(ca))["Source"]) + } + if m["$Type"] != domainmodel.OqlViewAssociationSource || m["Reference"] != "persistent_order" { + t.Errorf("cross-association Source = %v", m) + } + + // Control. + plain := dToM(serializeCrossAssociation(&domainmodel.CrossModuleAssociation{Name: "Plain"})) + if plain["Source"] != nil { + t.Errorf("a plain cross-association got Source = %v, want nil", plain["Source"]) + } +} From c6d0e258fa0aa92ffb2ea1b5dee6e92353358ae6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 21:35:09 +0000 Subject: [PATCH 18/18] fix(widgets): renumber the unroutable-child rule to MDL-WIDGET30, and guard the numbers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MDL-WIDGET29 was taken twice. This branch used it for the unroutable-child rule; main independently used it for the retired-spelling rule. Both were correct when written — nothing compared them until a rebase put both in the tree, and the number is only ambiguous once they meet. Renumbered this branch's rule to MDL-WIDGET30 and added the comparison as a test, because no amount of care at authoring time prevents a merge collision: the guard has to run where the collision exists, which is CI on the merge. The invariant is not "one file per id" — MDL-WIDGET21 is legitimately raised from two paths and is still one rule — so known pairs are listed and anything new fails, which is the moment to decide whether it is one rule or two. A second test keeps the numbers dense, so "the next free one" has an answer. Proven by reintroducing the collision: the guard names both files and says what to do. Its first version scanned for `RuleID: "…"` and missed MDL-WIDGET22, which is declared as a named constant — so it reported a gap that was not there. It now matches quoted literals, which is also what separates a declaration from a doc comment naming a neighbouring rule. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/overview-pages/SKILL.md | 2 +- cmd/mxcli/syntax/features_page.go | 4 +- docs/01-project/MDL_QUICK_REFERENCE.md | 4 +- ...datagrid-filter-block-dropped-silently.mdl | 4 +- mdl/executor/widget_rule_ids_test.go | 127 ++++++++++++++++++ mdl/executor/widget_unrouted_children.go | 4 +- mdl/executor/widget_unrouted_children_test.go | 4 +- 7 files changed, 138 insertions(+), 11 deletions(-) create mode 100644 mdl/executor/widget_rule_ids_test.go diff --git a/.claude/skills/mendix/overview-pages/SKILL.md b/.claude/skills/mendix/overview-pages/SKILL.md index 96f9220e8..db6417e6e 100644 --- a/.claude/skills/mendix/overview-pages/SKILL.md +++ b/.claude/skills/mendix/overview-pages/SKILL.md @@ -225,7 +225,7 @@ datagrid dg (...) { column colName (attribute: Name) { textfilter f1 } } -- ✅ gallery: the widget-wide filter bar, which the gallery calls `filter` gallery g (...) { filter f { textfilter f1 } } --- ❌ the gallery form on a data grid — MDL-WIDGET29 +-- ❌ the gallery form on a data grid — MDL-WIDGET30 datagrid dg (...) { column colName (attribute: Name) filter f { textfilter f1 } } ``` diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 28f2ddd0b..793ab9fae 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -133,7 +133,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "GALLERY g (...) { FILTER f { TEXTFILTER tf (Attribute: A) } }\n" + "-- A FILTER block written on a DATAGRID is not a column filter and not a\n" + "-- container the grid declares — it used to be dropped on write with no\n" + - "-- diagnostic, and is now refused (MDL-WIDGET29).\n\n" + + "-- diagnostic, and is now refused (MDL-WIDGET30).\n\n" + "-- Inputs\nTEXTBOX name (Label: 'L', Attribute: Attr)\nTEXTAREA | DATEPICKER | COMBOBOX | CHECKBOX | RADIOBUTTONS\n\n" + "-- Actions\nACTIONBUTTON name (Caption: 'C', Action: SAVE_CHANGES, ButtonStyle: Primary)\nLINKBUTTON name (Caption: 'C', Action: ...)\n\n" + "-- Display\nDYNAMICTEXT name (Content: 'Hello, {1}!', ContentParams: [{1} = Name])\nTITLE name (Content: 'Heading')\nIMAGE name (Image: 'Module.Collection.ImageName')\nIMAGE name (ImageType: imageUrl, ImageUrl: 'https://…')\n" + @@ -154,7 +154,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "-- DROPDOWN -> COMBOBOX\n" + "-- And three the executor refuses on BOTH engines, each with its own message:\n" + "-- STATICTEXT (writes Forms$Text, a type Mendix no longer has — the\n" + - "-- project could not be OPENED afterwards; MDL-WIDGET29.\n" + + "-- project could not be OPENED afterwards; MDL-WIDGET30.\n" + "-- Use DYNAMICTEXT with a literal Content.)\n" + "-- REFERENCESELECTOR (unsupported widget type)\n" + "-- LEGACYDATAGRID (use DATAGRID for the pluggable equivalent on Mendix 11+)", diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 895750e74..14e16eba2 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1340,8 +1340,8 @@ MDL uses explicit property declarations for pages: | Page CSS class / style | `Class: 'css-class', Style: 'css: rule'` | `(Title: 'Home', Class: 'container-fluid bg-light', Style: 'min-height: 100vh')` — the page's Appearance | | Page variables | `variables: { $name: type = 'expr' }` | `variables: { $show: boolean = 'true' }` | | Repeated widget entries | ` ( … )` **in the widget body** | A repeatable property (FileUploader `allowedFileFormats`, HTML Element `attributes`, a chart's `series`) is a block, never a property value. `attributes: [(attributeName: 'x')]` is **MDL-WIDGET27** — it used to check clean, exec, and vanish from storage. `describe widget -p app.mpr` lists the container keywords | -| Data grid 2 column filter | `column c (attribute: A) { textfilter f }` | **Inside the column's braces.** `column c (…) filter f { … }` is the GALLERY form — the grammar reads it as a column with no body plus a sibling `filter` widget, which the grid has nowhere to put; it used to be dropped on write and is now **MDL-WIDGET29**. A grid-wide filter bar is `controlbar`; a gallery spells that same slot `filter`. Match the filter to the column's type (String → `textfilter`, number → `numberfilter`, DateTime → `datefilter`, Enumeration → `dropdownfilter`, Boolean → none) | -| Widget with nowhere to go | any widget in a pluggable widget's body | A child matching no container, slot or `template` catch-all is **MDL-WIDGET29** at check time and refused by `exec`. `describe widget -p app.mpr` lists what the parent declares. Needs the parent's definition, so it is silent without `-p` | +| Data grid 2 column filter | `column c (attribute: A) { textfilter f }` | **Inside the column's braces.** `column c (…) filter f { … }` is the GALLERY form — the grammar reads it as a column with no body plus a sibling `filter` widget, which the grid has nowhere to put; it used to be dropped on write and is now **MDL-WIDGET30**. A grid-wide filter bar is `controlbar`; a gallery spells that same slot `filter`. Match the filter to the column's type (String → `textfilter`, number → `numberfilter`, DateTime → `datefilter`, Enumeration → `dropdownfilter`, Boolean → none) | +| Widget with nowhere to go | any widget in a pluggable widget's body | A child matching no container, slot or `template` catch-all is **MDL-WIDGET30** at check time and refused by `exec`. `describe widget -p app.mpr` lists what the parent declares. Needs the parent's definition, so it is silent without `-p` | | Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. **Body containers** names what the widget's body takes, and for an object list the widgets-typed slots *inside one item* plus the widget types that route into each — that is where `column … { textfilter }` is spelled out. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | diff --git a/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl b/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl index 12a47c523..b66ce5f92 100644 --- a/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl +++ b/mdl-examples/bug-tests/datagrid-filter-block-dropped-silently.mdl @@ -24,12 +24,12 @@ -- leftovers were discarded. Not datagrid-specific: one `continue` per pass, -- shared by every pluggable widget. -- --- Fix: MDL-WIDGET29 at check time and a refusal in buildPluggable at exec time, +-- Fix: MDL-WIDGET30 at check time and a refusal in buildPluggable at exec time, -- both from unroutedPluggableChildren() so they cannot disagree. The message -- names the spelling THIS widget uses for the same slot, since an author who -- copied a gallery example needs `controlbar`, not a list to search. -- --- This repro is a plain .mdl, not .fail.mdl, on purpose: MDL-WIDGET29 needs the +-- This repro is a plain .mdl, not .fail.mdl, on purpose: MDL-WIDGET30 needs the -- parent widget's DEFINITION, which comes from the project's installed .mpk, and -- `make check-mdl` runs `check` with no project (see the Makefile's note on -- #891/#892). The refusal is covered by widget_unrouted_children_test.go; this diff --git a/mdl/executor/widget_rule_ids_test.go b/mdl/executor/widget_rule_ids_test.go new file mode 100644 index 000000000..08cf11219 --- /dev/null +++ b/mdl/executor/widget_rule_ids_test.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" +) + +// MDL-WIDGET rule numbers are handed out by hand, from no registry, by grepping +// for the highest one in use. That has now failed twice: +// +// - once at authoring time, picking 27 for a rule that already existed; +// - once at MERGE time, where two branches independently took 29 — one for a +// retired-spelling rule, one for an unroutable child — and each was correct +// in isolation. Nothing compared them until a rebase put both in the tree. +// +// The second is the one that matters, because no amount of care at authoring +// time prevents it. This test does the comparison, so the collision fails on the +// merge that creates it rather than shipping as two rules answering to one id. +// +// The invariant is NOT "one file per id": a rule legitimately raised from two +// places stays one rule. So known pairs are listed, and anything new fails — +// which is exactly the moment to check whether it is one rule or two. +var widgetRuleIDsRaisedFromSeveralFiles = map[string][]string{ + // One rule about a property that is hidden under the current configuration, + // raised from the content-params path and the editability path. + "MDL-WIDGET21": {"validate_widget_contentparams.go", "validate_widget_editability.go"}, +} + +// Matched as a QUOTED literal rather than after `RuleID:`, because an id is +// also declared as a named constant (imageSourceRule = "MDL-WIDGET22") — the +// first version of this guard missed that one and reported a gap that was not +// there. Quoting is what separates a declaration from a doc comment mentioning +// a neighbouring rule. +var widgetRuleIDRe = regexp.MustCompile(`"(MDL-WIDGET\d+)"`) + +func TestWidgetRuleIDsAreNotReused(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + byID := map[string]map[string]bool{} + for _, f := range files { + if strings.HasSuffix(f, "_test.go") { + continue + } + src, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + for _, m := range widgetRuleIDRe.FindAllStringSubmatch(string(src), -1) { + if byID[m[1]] == nil { + byID[m[1]] = map[string]bool{} + } + byID[m[1]][f] = true + } + } + if len(byID) == 0 { + t.Fatal("no MDL-WIDGET rule ids found — this guard would pass vacuously") + } + + for id, files := range byID { + if len(files) < 2 { + continue + } + var got []string + for f := range files { + got = append(got, f) + } + sort.Strings(got) + want := append([]string(nil), widgetRuleIDsRaisedFromSeveralFiles[id]...) + sort.Strings(want) + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Errorf("%s is raised from %v.\n"+ + "If those are TWO different rules, one of them needs a new number — take the next "+ + "free one after the highest in use, and check again after rebasing, because another "+ + "branch may have taken it meanwhile.\n"+ + "If it is ONE rule raised from two places, add it to "+ + "widgetRuleIDsRaisedFromSeveralFiles.", id, got) + } + } +} + +// The numbers are also expected to be dense, so "the next free one" is a +// question with an answer. A gap means a rule was removed without its number +// being reused, which is fine but worth stating deliberately rather than +// leaving the next author to guess. +func TestWidgetRuleIDsHaveNoGaps(t *testing.T) { + files, _ := filepath.Glob("*.go") + seen := map[int]bool{} + max := 0 + numRe := regexp.MustCompile(`MDL-WIDGET(\d+)`) + for _, f := range files { + if strings.HasSuffix(f, "_test.go") { + continue + } + src, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + for _, m := range widgetRuleIDRe.FindAllStringSubmatch(string(src), -1) { + n := 0 + for _, d := range numRe.FindStringSubmatch(m[1])[1] { + n = n*10 + int(d-'0') + } + seen[n] = true + if n > max { + max = n + } + } + } + var missing []int + for i := 1; i < max; i++ { + if !seen[i] { + missing = append(missing, i) + } + } + if len(missing) > 0 { + t.Errorf("MDL-WIDGET numbers are not dense: %v missing below %d. "+ + "Reuse the lowest free number, or note here why it is retired.", missing, max) + } +} diff --git a/mdl/executor/widget_unrouted_children.go b/mdl/executor/widget_unrouted_children.go index 49e4a2bac..6e96c56e0 100644 --- a/mdl/executor/widget_unrouted_children.go +++ b/mdl/executor/widget_unrouted_children.go @@ -144,7 +144,7 @@ func sameSlotUnderAnotherName(def *WidgetDefinition, keyword string) string { return "" } -// validateUnroutedChildren is the check-time half (MDL-WIDGET29). It sits beside +// validateUnroutedChildren is the check-time half (MDL-WIDGET30). It sits beside // MDL-WIDGET26, which covers the neighbouring case: a container KEYWORD (`group`, // `series` — words that are not widgets at all) under a parent that does not // declare it. This one covers a real WIDGET in the same position, which @@ -153,7 +153,7 @@ func validateUnroutedChildren(w *ast.WidgetV3, def *WidgetDefinition, locationPr var out []linter.Violation for _, child := range unroutedPluggableChildren(def, w) { out = append(out, linter.Violation{ - RuleID: "MDL-WIDGET29", + RuleID: "MDL-WIDGET30", Severity: linter.SeverityError, Message: fmt.Sprintf("%s: %s", locationPrefix, unroutedChildMessage(def, child)), Suggestion: "move it into one of the parent's containers, or out of the widget's body", diff --git a/mdl/executor/widget_unrouted_children_test.go b/mdl/executor/widget_unrouted_children_test.go index f05ce4392..4c5b9c2bb 100644 --- a/mdl/executor/widget_unrouted_children_test.go +++ b/mdl/executor/widget_unrouted_children_test.go @@ -207,8 +207,8 @@ func TestUnroutedViolationIsAnError(t *testing.T) { if len(v) != 1 { t.Fatalf("got %d violations, want 1", len(v)) } - if v[0].RuleID != "MDL-WIDGET29" { - t.Errorf("RuleID = %s, want MDL-WIDGET29", v[0].RuleID) + if v[0].RuleID != "MDL-WIDGET30" { + t.Errorf("RuleID = %s, want MDL-WIDGET30", v[0].RuleID) } if v[0].Severity != linter.SeverityError { t.Errorf("Severity = %s, want error — a dropped widget is not a style note", v[0].Severity)