Skip to content

fix(export): load sysml-toolkit API JSON exports and accept JSON model files - #876

Merged
HuiJun merged 11 commits into
developfrom
fix/toolkit-json-import
Oct 4, 2026
Merged

HuiJun merged 11 commits into
developfrom
fix/toolkit-json-import

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

What and why

API element-form JSON exported by sysml-toolkit v0.10.0 (sysmlv2 convert … --to full-json) failed to load through sysml -convert sysml -from api-json. This change fixes the three measured failures, cuts the cost of a large import, and lets a .json model be loaded anywhere a model file is accepted.

  • FlowEnd (drones). A FlowEnd owned through EndFeatureMembership is now folded into the owning flow's from … to … head and is never written as a standalone declaration. The toolkit's shape, a FlowFeature ReferenceUsage owned through a FeatureMembership with one implied redefinition, is accepted alongside the existing forms. A flow end that declares a name or annotations, or owns content the head cannot carry, is refused. Other flow-related metaclasses in the exports were checked: FlowUsage, PayloadFeature, EndFeatureMembership and ReferenceSubsetting. The exports contain no ItemFlow, ItemFlowEnd or SuccessionItemFlow.
  • Transitions (pumps). The unparsable notation was … accept Pumps::Boost do in attribute then …: the toolkit's implicit parameters (transitionLinkSource, the trigger payload) were read as the transition's effect. Trigger, guard and effect are now read by their TransitionFeatureMembership kind. A ParameterMembership is never an effect. The source parameter is recognised by its implied redefinition of Actions::TransitionAction::transitionLinkSource. Guards stated through a guard membership or sysml:guardExpression are written as if …; before this change they were silently dropped. Names written inside a trigger resolve from the transition's scope.
  • Library identities (Apollo). The Apollo error did not come from library id assignment. CoSMAPackage and two other packages are Apollo's own library packages. Removing the toolkit's synthetic root namespace left them unowned, and they were then mistaken for standard-library reference stubs. A library package that owns content is now a declaration. For real stubs whose id is not in the bundled catalog, the reader falls back to the stated qualifiedName, then to declaredName plus the owner chain the graph states, then to an imported membership's target. The match's metaclass must be compatible. A fallback match is written by its bundled identity and reports a warning naming both ids. An unmatched stub is refused with its id and name. Across the three exports, 205 of the 221 element ids that are referenced but not contained already match the bundled catalog by id; the rest are derived result links and one membership import.
  • Satisfy subjects and return parameters. A satisfy … by a.b subject is written back as by a.b. Its value is the FeatureReferenceExpression whose referent is an owned chain feature (SysML.xtext SatisfactionReferenceExpression). A toolkit return :> x = …, which is a ReferenceUsage, is no longer written as return attribute ….
  • Performance. The dominant cost was quadratic: member ordering re-listed every subject in the graph once per owner, about 50 GB of allocations on Apollo. Subjects are now bucketed by owner once, and repeated full-graph scans (reference subsettings, port definitions) are indexed. Memory: API JSON is streamed into a GraphBuilder, which de-duplicates once at build time. The import's own graph is rewritten in place (Graph.RewriteTriples) instead of copied at each normalisation step, and rdf.Graph caches subject order and indexes objects by triple position. Public export.ToSysML/ToSysMLWarn still convert a copy and leave the caller's graph unchanged; only the new export.APIJSONToSysML, which owns the graph it reads, rewrites in place.
  • JSON wherever a model loads. An explicitly named .json file is converted to SysML before parsing in sysml -validate, sysml file loading and the REPL, and gRPC ParseFile/ParseSources file paths. This covers Python opensysml.load("x.json"), which goes through ParseFile. Conversion warnings are printed on the CLI and REPL. Over gRPC they are returned as warning diagnostics tagged with the JSON path and no range. Directory and glob expansion still collect only .sysml/.kerml. source.KindOf is unchanged; the converting call sites set the SysML kind themselves.

Overlap with #865 (metamodel regeneration to 20250201): this branch merges develop after #865 and does not touch internal/translate/rdf/ontology/table.go or its generator. Every metaclass and predicate name it hard-codes (the flow constants in internal/translate/export/rdf_normative.go, transitionLinkSource, and the rest) exists in the regenerated table.

Measurements

The command is sysml -convert sysml -from api-json <file>, measured with /usr/bin/time -v. The "After" column was measured on the head merged with develop after #865. The exports were made by sysml-toolkit v0.10.0 with --lib internal/workspace/libs/stdlib.

Input Before (develop) After
sysml-sdk drones.sysml 0.20 s, 93 MB max RSS; fails (FlowEnd has no notation) 0.22 s, 96 MB; converts and validates
sysml-sdk pumps.sysml 0.15 s, 90 MB; fails (written notation does not parse) 0.17 s, 91 MB; converts and validates
Apollo 11, all 28 files, 92 MB JSON 120.7 s, 1.83 GB; fails (library element id) 9.8 s, 0.87 GB; converts and validates

Apollo's validation reports no errors and three warnings that come from the model itself: an incommensurable-quantity addition, an unbound naturalLogarithm parameter, and a variable multiplicity on subfunctions[*].

Fidelity was checked by re-exporting the converted notation with the toolkit and comparing it with the original export. Drones (739 elements) and pumps (475) match in element count, count per @type and the set of qualified names. Apollo matches all 2,335 qualified names and every per-type count except Namespace 28 → 1. That difference comes from writing one document instead of 28 files: the toolkit run over the 28 files concatenated into one shows the same single difference. The 92 MB Apollo export is not committed.

Specification basis

  • Flow ends: SysML v2 FlowEnd/FlowFeature (SysML.xtext FlowEnd, FlowEndMember); a flow end is written in the flow head's from … to ….
  • Transitions: TransitionFeatureMembership kind trigger/guard/effect (SysML v2 § 8.3.17.9, SysML.xtext TriggerActionMember, GuardExpressionMember, EffectBehaviorMember).
  • Satisfy subject: SysML.xtext SatisfactionSubjectMember / SatisfactionReferenceExpression / FeatureChainMember.
  • Return parameters: a return UsageElement with no kind keyword is a ReferenceUsage (SysML-textual-bnf DefaultReferenceUsage).

No row in docs/project/spec-compliance.md moves. docs/reference/rdf-mapping.md documents the library-identity fallback, flow-end folding, transition guard/effect reading, satisfy-subject chains and JSON model loading.

How it was verified

  • New toolkit-style fixtures in tests/export/testdata/interchange/ (flow_ends, transitions, library_identity, library_unmatched, successions, satisfy_by), each a small .sysml source, its toolkit full.json and the expected notation. They are exercised by tests/export/toolkit_json_import_test.go, with unit tests in internal/translate/export and internal/translate/convert.
  • JSON loading: tests in cmd/sysml (-validate, -convert warning on stderr), internal/frontend/repl (explicit .json loads with its warning; a directory ignores .json) and internal/frontend/grpc (ParseFile returns the root and a path-tagged warning that is also served from the cache; an unconvertible file is InvalidArgument).
  • A test asserts that ToSysML leaves a Turtle-read graph's triples unchanged.
  • Converted .json documents keep the SysML source kind through workspace indexing, analysis, records, Update and REPL reparses (tests in internal/workspace/model and internal/frontend/repl). A named flow end is refused rather than folded.
  • The RDF corpus round trip with both corpus require variables set gives 358 files: 351 stable, 7 whitespace-only (the same seven as on develop), 0 graph-diff, 0 unwritable, 0 unparseable, 0 refused. No ratchet moved, and the training corpus is clean.
  • On the merged head: go build ./..., go vet ./..., gofmt -l ., make lint, go test ./..., go test -race ./internal/translate/... ./tests/export/..., go test for cmd/sysml, internal/frontend/grpc and internal/frontend/repl, make docs-check, and python3 scripts/changelog.py check.
  • No existing golden changed.

Checklist

  • make test and make lint pass locally (the packages touched are listed above; CI runs the full suite)
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (see CONTRIBUTING.md)
  • Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (no gate count moved)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

devin-ai-integration Bot and others added 4 commits October 3, 2026 20:39
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration Bot and others added 2 commits October 4, 2026 02:15
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 4, 2026 03:23
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits October 4, 2026 04:37
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@HuiJun
HuiJun merged commit c5966be into develop Oct 4, 2026
24 checks passed
@HuiJun
HuiJun deleted the fix/toolkit-json-import branch October 4, 2026 23:39
@devin-ai-integration devin-ai-integration Bot mentioned this pull request Oct 5, 2026
6 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant